Hypothesis: Reuse of End-User Code is an Adoption Process

Christopher ScaffidiCMU ISR SSSG Apr 20, 2009

22

End-user programmers (EUPs) often reuse programs, code, and other computational

artifacts.• Code created by other EUPs

○ Web macros○ Spreadsheets○ Matlab scripts○ JavaScript and HTML○ Agentsheet images and rules

• Code created by professional programmers (PPs)○ JavaScript interpreter○ Spreadsheet evaluation environment○ Other components designed by PPs for reuse○ Code snippets / examples provided by PPs

All of which I’ll refer to as reuse of code in this talk.I am aware that this might be a misnomer… discuss later.

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

33

Systems supporting reuse of EUP-created code

are based on…• What seems like a “natural choice” of features

○ Web macro wiki (Little 2007)

• What features proved successful in prior systems○ Agentsheet repository (Rausch 1996)

○ Tope repository (Scaffidi 2009)

• Reflections about the process of reuse (Biggerstaff 1987)

○ Spreadsheet repository (Walpole 1997)

• Empirically trying out features○ Matlab FileExchange (Gulley 2006)

Can we know, in advance, what to build?

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

44

Needed: a model describing the reuse of EUPs’ code.

• What is a good empirical model for reuse of EUP code?○ Key steps involved in reuse?○ Key considerations for judging if reuse is successful?○ Key factors driving successful reuse?

• Such a model could help…○ Designers of EUP systems○ Organizations that use EUP systems

• Eg: suppose the model says that EUPs usually…○ Want to try out code before reusing it○ Have more success in reuse if code is “try-out-able”

then EUP system designers would know that they should work very hard to support “try-out-ability”

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

55

Known: Adoption models accurately describe

many (re)use processes.• “Adoption” = “a decision to make full use of an innovation

as the best course of action available” (Rodgers 1995)

• Adoption models have accurately described (re)use of…○ Hybrid strains of corn seed by Iowa farmers (Rodgers 1995)

○ CASE, structured programming, RDBMSs by PPs (Fichman 1993)

○ A new SE methodology by PPs (Riemenschneider 2002)

○ Computers by end users (Scaffidi 2005)

○ Information systems by end users (Delone 2003)

○ Spreadsheet tools by EUPs (Brynjolfsson 1996)

… and others discussed later.

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

66

Hypothesis: Adoption models accurately describe

EUPs’ reuse of EUP-created code.Since adoption models work well for many kinds of reuse

by end users, how about reuse of EUP code by EUPs?

• Reasons to question this hypothesis:○ EUPs focus on work… is reuse a “decision” aimed at attaining a

“best” course of action?○ EUPs rarely design for reuse… is their code intelligible enough

to support rational decisions?

• Other theories might help, but lack the right focus:○ Agency theory – incentives, delegation, decision structure○ Post-rational behavior models – motivation, trust, activities○ Behavioral economics – emotions, framing, cognitive limitation

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

77

Outline

• Introduction

• Models from adoption theories○ TAM with external variables○ Diffusion models○ Elaboration from economics

• Hypothesis○ (Future) evaluation○ Implications

IntroductionIntroduction • Models • Models • Hypothesis• Hypothesis

88

Basic Technology Acceptance Model (TAM)

(Davis 1985) Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

99

TAMplus external variables

(Lee 2003) Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1010

Diffusion models –Rate of adoption

(Rodgers 1995) Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1111

Diffusion models –Cumulative adoption

(Scaffidi 2005; Rodgers 1995) Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1212

Key elements in diffusion models

• Innovation’s properties○ Relative advantage○ Compatibility○ Complexity○ Trialability○ Observability

• Communication channels○ Heterophily

• Time○ Innovators-early adopters-

early majority-late majority-laggards

○ Knowledge-persuasion-decision-assimilation-confirmation

• Social system○ Structures (eg: hierarchy)○ Coordination (in decisions)○ Norms

(Rodgers 1995) Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1313

Prior successesof adoption models

• Adoption by PPs○ CASE, structured programming, RDBMSs (Fichman 1993)

○ Components (Tahvildari 2004) (Smith 2004)

○ SE methods/processes (*Riemenschneider 2002) (Pfleeger 2000)

• Adoption by end users○ Computers (*Scaffidi 2005)

○ Information systems (*Delone 2003) (*Lee 2003)

○ Spreadsheet tools (*Brynjolfsson 1996)

○ EUP tool enhancements (Kienle 2002)

○ Web browsers (*Agarwal 1997)

○ Communication technologies (*Forman 2005)

○ Online services (*Parthasarathy 1998)

* = quantitative model (ie: not just a qualitative adoption approach)

Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1414

Particularly interesting findingsrelated to adoption by PPs and EUPs

• Key factors limiting reuse of PP code by PPs○ Prior technology drag – affects compatibility w/ existing network○ Irreversibility of investments – affects trialability○ Limited sponsorship – calls for more than “communication”○ High expectations – sets a high bar for each adoption stage(Fichman 1993)

• Key factors limiting reuse of PP code by EUPs○ Smaller organization size – less risk capacity, network effects○ More organization hierarchy – limits innovation’s impact(Forman 2005)

Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1515

Particularly interesting findingsrelated to adoption by PPs and EUPs

• Key factors limiting assimilation of PP code by PPs○ Knowledge barriers – complexity + incompatibility with training○ Unrealistic expectations about positive network effects(Fichman 1999)

• Key factors raising discontinuance of PP code by EUPs○ Later adopters lack resources to make full use of innovation○ People who encounter trouble with incompatibility lose benefits(Parthasarathy 1998)

Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis

1616

Is reuse of EUPs’ code an

adoptionprocess?

1717

Is EUP reuse of EUPs’ codean adoption process?

• When an EUP reuses another EUP’s code, does he go through the typical stages of adoption?

○ Knowledge-Persuasion-Decision-Assimilation-Confirmation

• Contextual inquiry○ Identify reused code

• Uncover source, history, expectations, communication, sponsorship• Did source ever provide other code? What happened?

○ Any sign of assimilation & confirmation for users’ own code?• Look for evidence of “not-invented-here”

• Complement with survey?○ Possibly a survey about spreadsheets, modeled after (Hall 1996)

○ Alternatively, integrate with CoScripter survey (next slide)

Introduction Introduction • Models • Models • • HypothesisHypothesis

1818

Is EUP reuse of EUPs’ codefundamentally an adoption process?

• How well is the rate of EUP code adoption described by diffusion models?

○ Logistic increase in adoption?○ Do users fit early/late adopter stereotypes?

• Analyze logs of CoScripter web macro repository○ Non-linear regression of usage to logistic curve○ Use chained analysis to check how well users fit stereotypes

• Complement with survey of users within IBM?○ Group heavy users of popular macros, based on adoption date○ Survey about key characteristics of stereotypes

Introduction Introduction • Models • Models • • HypothesisHypothesis

1919

Is EUP reuse of EUPs’ codefundamentally an adoption process?

• What characteristics of EUP code affect reuse?○ Consistent with adoption models?○ Which characteristics are more/less crucial?

• Return to last year’s analysis of CoScripter logs○ Are the identified predictors consistent with adoption models?○ Extend analysis, using additional adoption-motivated variables?○ Repeat analysis using regression or factor analysis?○ Do later adopters & lower assimilators tend to discontinue?

• Complement with similar study of Matlab logs?

Introduction Introduction • Models • Models • • HypothesisHypothesis

2020

Tie together resultsinto a coherent model

• Key factors in decision, assimilation, discontinuance○ Properties of EUP code○ Properties of communication channels and social structure○ Properties of EUPs

• Perhaps something like…

Introduction Introduction • Models • Models • • HypothesisHypothesis

Factor 1

Factor 2

Factor 3

… Assimilation

Decision to (re)use

Discontinuance

Perceived ease-of-use

Perceived usefulness

(The interestingcascading arrowson the right are consistent with empirical researchdiscussed earlier.)

2121

Research opportunities if EUP reuse of EUPs’ code is an adoption

process…• Helping EUPs…

○ To create code with key characteristics leading to adoption (eg: compatibility, observability, trialability, …)

○ To provide constructive sponsorship without sacrificing primary job responsibilities (ie: how to sponsor while remaining an EUP)

• Helping organizations…○ Manage heterophily in communication channels

• Possibly through creative use of limited hierarchy○ Accelerate adoption cycle

• Knowledge-persuasion-decision-assimilation-confirmation○ Learn from discontinuance

Introduction Introduction • Models • Models • • HypothesisHypothesis

2222

Thank You…

• To Mary Shaw for early feedback on this work

• To EUSES and NSF for funding

• To you for your attention, insights, and feedback

2323

References (1)

Agarwal 1997 - R. Agarwal and J. Prasad. The Role of Innovation Characteristics and Perceived Voluntariness in the Acceptance of Information Technologies. Decision Sciences, Vol. 28, No. 3, 1997, pp. 557-582.

Biggerstaff 1987 - T. Biggerstaff and C. Richter. Reusability Framework, Assessment, and Directions, IEEE Software (4), 2, March 1987, 41-49.

Brynjolfsson 1996 - E. Brynjolfsson and C. Kemerer. Network Externalities in Microcomputer Software: An Econometric Analysis of the Spreadsheet Market. Management Science, 1996, pp. 1627-1647.

Davis 1980 - F. Davis. A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results, PhD. Thesis, Sloan School of Management, Massachusetts Institute of Technology, 1985.

Delone 2003 - W. Delone and E. McLean. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, Vol. 19, No. 4, 2003, pp. 9-30.

Fichman 1993 - R. Fichman and C. Kemerer. Adoption of Software Engineering Process Innovations: The Case of Object Orientation. Sloan Management Review, Vol. 34, 1993, pp. 7-22.

Fichman 1999 - R. Fichman and C. Kemerer. The Illusory Diffusion of Innovation: An Examination of Assimilation Gaps. Information Systems Research, Vol. 10, No. 3, 1999, pp. 255-275.

Forman 2005 - C. Forman and A. Goldfarb. Diffusion of Information and Communication Technologies to Businesses, Handbook of Economics and Information Systems, Elsevier Science, 2005.

Gulley 2006 - N. Gulley. Improving the Quality of Contributed Software and the MATLAB File Exchange, 2nd Workshop on End User Software Eng., 2006, 8-9.

Hall 1996 - Hall, J. A Risk and Control-Oriented Study of the Practices of Spreadsheet Application Developers. Proc. 29th Hawaii Intl. Conf. System Sciences, 1996, 364-373.

2424

References (2)

Kienle 2002 - H. Kienle, A. Weber, J. Jahnke, and H. Muller. Tackling the Adoption Problem of Domain-Specific Visual Languages. 2nd Domain-Specific Modeling Languages Workshop At OOPSLA 2002, 2002, pp. 77-88.

Lee 2003 - Y. Lee, K. Kozar, and K. Larsen. The Technology Acceptance Model: Past, Present, and Future. Communications of the Association for Information Systems (Volume 12, Article 50), Vol. 752, No. 50, pp. 752-780.

Little 2007 - G. Little, T. Lau, A. Cypher, and J. Lin. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2007, pp. 943-946.

Parthasarathy 1998 - M. Parthasarathy and A. Bhattacherjee. Understanding Post-Adoption Behavior in the Context of Online Services. Information Systems Research, Vol. 9, 1998, pp. 362-379.

Pfleeger 2000 - S. Pfleeger and W. Menezes. Marketing Technology To Software Practitioners. IEEE Software, Vol. 17, No. 1, 2000, pp. 27-33.

Rausch 1996 - M. Rausch. The Agent Repository-Supporting Collaborative Contextualized Learning with a Medium for Indirect Communication, Master's Thesis, Department of Computer Science, University of Colorado, 1996.

Riemenschneider 2002 - C. Riemenschneider, B. Hardgrave, and F. Davis. Explaining Software Developer Acceptance of Methodologies: A Comparison of Five Theoretical Models. IEEE Transactions on Software Engineering, Vol. 28, No. 12, 2002, pp. 1135-1145.

Rodgers 1995 - Rogers, E. Diffusion of Innovation, 4th edition, Simon & Schuster, Inc., New York, NY, 1995.

2525

References (3)

Scaffidi 2005 - C. Scaffidi, M. Shaw, and B. Myers. Estimating the Numbers of End Users and End User Programmers. Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2005), Dallas, TX, September 2005, pp. 207-214.

Scaffidi 2009 - C. Scaffidi, Topes: Enabling End-User Programmers to Validate and Reformat Data, PhD thesis, Technical Report CMU-ISR-09-105, School of Computer Science, Carnegie Mellon University, 2009.

Smith 2004 - D. Smith, E. Morris, D. Carney. Adoption Centric Problems in the Context of System of Systems Interoperability, 4th International Workshop on Adoption-Centric Software Engineering, pp. 63-68.

Tahvildari 2004 - L. Tahvildari. Testing Challenges in Adoption of Component-Based Software. 4th International Workshop on Adoption-Centric Software Engineering, pp. 21-25.

Walpole 1997 - R. Walpole and M. Burnett. Supporting Reuse of Evolving Visual Code, 1997 Symp. Visual Lang., 1997, 68-75.

2626

Updated Delone & McLean modelof information system (IS) success

(Delone 2003)

you built the right IS

you built the IS to spec

you give good user support

Introduction Introduction • • ModelsModels • Hypothesis• Hypothesis