Algorithm Adoption Model: What factors lead to algorithm

adoption and use by decision-makers

JOEL DAVIS

JUNE 2020

A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF BUSINESS

AT THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF

DOCTOR OF BUSINESS ADMINISTRATION

UNIVERSITY OF FLORIDA

2020

© 2020 Joel Davis

DEDICATION

To my incredible wife, Sonia Davis, my heartfelt thank you. I love you.

3

ACKNOWLEDGMENTS

I would like to express my most profound appreciation to my dissertation chair Dr. Philip

Podsakoff. Thank you for spending the time to guide me through this patiently. This doctoral

program has been a fantastic journey.

I very much appreciate the support of Revenue Management Solutions, and the leadership team

of John Oakes, Olivier Rougie, Sebastian Fernandez, Mark Kuperman, and Jana Zschieschang

for supporting me through this endeavor. Thank you.

I would like to thank Angie Woodham, who shepherds us through the complexities of earning a

doctorate. Thanks for your time and the extra effort you put into making this a great program and

a great experience.

Finally, I would like to thank my fellow cohort members for their support and friendship. Thank

you for the words of encouragement, advice, and most of all, for listening. I am proud to be

among you.

4

Abstract

There is a long academic history describing decision-makers avoiding or under-weighting

algorithmic advice. Recent experimental studies have shown that decision makers reject using algorithms

under various conditions. Despite this, the management field has not developed the necessary theoretical

understanding and conceptual definitions that underlie algorithm adoption. In this study, semi-structured

interviews were conducted with thirty business professionals across a variety of industry backgrounds, all

of whom have had multiple occasions to consider using the advice of an algorithm to make or improve a

decision. Through an analysis of the interviews and an integration of the literature on technology adoption

and trust in automation, an algorithm adoption model (AAM) construct is defined and developed.

Algorithm adoption consists of 4 sub-dimensions: (1) input trust, (2) output trust, (3) algorithm

provenance, and (4) understandability. This study first provides clarity and definitions of algorithms and

trust through a review of the literature. It considers how factors related to technology adoption,

technology diffusion, human-human trust, and human-automation trust relate to the concept of trusting

algorithms. The conceptual positioning of the algorithm adoption model vs. existing models of

technology adoption and automation trust, and the similarities and differences between these models are

explored and discussed. Finally, the benefits, barriers, and risks of adopting algorithms into decision-

making are examined.

5

Introduction

Increasingly, decisions that were in the past only made by humans are now strongly influenced by

one or more algorithms. This evolution and growth in algorithm-related decision-making is not without

some casualties. In some cases, algorithms are substitutes for human decision-makers, and in other cases

the algorithm compliments a decision maker. As any introductory economics textbook explains, some

compliments will become more valuable as cognitive algorithms improve in performance, and some

substitutes will become less valuable. As noted by Agrawal, Gans, and Goldfarb (2018), London cab

drivers are an excellent example of the latter point. Before the introduction of sophisticated wayfinding

algorithms like Google Maps or Waze, prospective London cab drivers spent years learning the roads,

locations, and best routes through London. This daunting task ended in a test to ensure that they were able

to meet the incredibly high standards of those that drove before them. This test created a very high barrier

of entry that made the career a lucrative one. However, the introduction of step-by-step and turn-by-turn

algorithm-based solutions meant almost anyone with a car could compete with London Cabbies and

turned a decades-old tradition and industry on its head. Given the fundamental change, how do cab

drivers in London react? Do they change their standard practice and adopt the algorithm's advice, or do

they continue with their traditional approach? It is easy to see that the decision in many cases is not a

dichotomous one; in our example, the drivers can use none, some, or all the advice. It is also easy to

surmise that how much advice an individual is prepared to take from an algorithm is related to his/her

level of expertise and willingness to accept advice in general. This constant push of capabilities

enhancement, compliments, and substitutions are poised to have an enormous effect on the way we work

and live our lives in the future (Agrawal et al., 2018; Malone, 2018).

Given the importance of algorithms in our economy, and in our professional and personal lives, it

is critical to understand how human decision-makers accept or reject the outputs of algorithms. Dietvorst,

Simmons, and Massey (2015) showed that people are more apt to lose confidence in an algorithm after

seeing it err than when a human forecaster makes similar errors. Interestingly, even people who see an

algorithm outperform the human forecaster are less likely to follow its recommendations. This pattern of

6

findings builds on the seminal work of Meehl (1954), who found that linear models outperformed

psychologists' clinical approaches, even when the models were imperfect (Dawes, 1979). This research

has created an entire field of scholarly work. Much of this work supports the finding that algorithmic

combinations, even naive ones, are superior to human judgement. For example, a meta-analysis reported

by Grove, Zald, Lebow, Snitz, and Nelson (2000) found that, with few exceptions, mechanical prediction

techniques are more accurate than clinical predictions. This led these authors to conclude that, "There

seem, then, to be no barriers to a general preference for mechanical prediction where an appropriate

mechanical algorithm is available" (Grove et al., p. 26). Similar results were reported by Kuncel, Klieger,

Connelly, and Ones (2013), who showed that even when the decision-makers are experts, mechanical

combinations significantly outperformed holistic/human combinations.

So why do users adopt or fail to adopt algorithmic advice? Existing academic literature often

compares a choice to be made between accepting an algorithm’s advice or using an individual's own

judgment (Dawes, Faust, and Meehl, 1989; Dietvorst et al., 2015; Meehl, 1954). But, what happens when

the choice is not between one's self and an algorithm but rather between algorithms? Or when the

algorithm offers advice first, and then the decision-maker decides to accept or reject that advice? One

obvious avenue of research that focuses on answering these questions it the literature on trust in

automation (Lee and See, 2004; Muir, 1987). This research, however, tends to focus on specific and low-

level automation tasks such as automated factory monitoring system; not tasks that require higher levels

of cognitive work on the part of the decision-maker (Prahl and Van Swol, 2017).

Algorithm adoption is different and perhaps more complicated than human to human advice

adoption. One reason for this is that algorithms can lack accessibility. Even simple algorithms may be

hard for decision-makers to interpret and understand (Diakopoulos, 2014). This lack of accessibility may

have the effect of diminishing the decision-makers' trust in the algorithm, leading to lower rates of

adoption (Muir, 1987). Other factors of trust also play a significant role. Prior research in automation has

shown that credibility (i.e., will the machine work), is a relevant factor in automation trust. Extending this

trust effect from automation to algorithms makes sense given the findings of algorithm aversion reported

7

by Dietvorst et al. (2015) and Prahl and Van Swol (2017). Furthermore, decision-makers' prior

experiences with algorithms is likely to affect their adoption processes (McKnight, Choudhury, &

Kacmar, 2002; McKnight, Cummings, & Chervany, 1998).

Finally, it is important to note that we are not suggesting that the solution to improving human

decision-making process is simply to get decision-makers to adopt more algorithms. In some cases,

algorithms express severe bias, either because of limitations of the programming or because of the data

they are trained on (Kitchin, 2017). If individuals adopt algorithms based on many complex factors,

research such as this has a significant academic and practical interest. An essential human judgment task

is to understand the causes and remedies for this bias, in both the algorithms they use and how the

decision-makers adopt and use them. An important societal goal is understanding how and when

individuals and firms incorporate algorithms into their decision-making process, and how this can be

improved to facilitate better decision- making by reducing both human and machine bias.

Within the context of the above discussion, the purpose of the current research is to develop a

better understanding of the algorithm adoption process. In order to accomplish this goal, the following

research questions guided this study:

1. What is the meaning of the term "algorithm" and "algorithm adoption"? More specifically,

what are the defining properties of these constructs?

2. What are the factors that lead to Algorithm Adoption by decision-makers?

3. To what extent does trust in an algorithm's inputs and outputs relate to Algorithm Adoption?

4. How is Algorithm Adoption similar/different from other, related constructs such as the

Technology Acceptance Model (TAM) (Davis, 1989)?

5. What are the antecedents of Algorithm Adoption (such as personality traits, motives, the

nature of the task the algorithm is being applied to, organizational factors, etc.)?

6. What are the benefits and barriers to individuals/firms of adopting algorithms in decision-

making?

8

What is an Algorithm?

Before examining the literature, it is important to clarify what is meant by an algorithm.

Algorithms are often part of more extensive decision-making programs or tools used by organizations and

individuals. These types of interconnections make studying algorithms somewhat challenging, as their

delivery depends on other factors. Studies have shown, for example, that the interface a program uses can

impact a user’s adoption of that system. As an example, Schwartz and Cohen (2004) conducted a study

utilizing forecasting systems and found that the speed and content of the interface was an important factor

in determining the systems use. Given this, how do we know what is being evaluated, the program or

technology delivering the algorithm, or the algorithm itself? First, it is essential to realize that although

algorithms are often embedded in other technological solutions, they differ in significant ways. These

differences make the study of algorithms as an entity a critical endeavor. Indeed, according to Dourish

(2016):

Algorithms and programs are different entities, both conceptually and technically. Programs may

embody or implement algorithms (correctly or incorrectly), but, as I will elaborate, programs are

both more than algorithms (in the sense that programs include non-algorithmic material) and less

than algorithms (in the sense that algorithms are free of the material constraints implied by

reduction to particular implementations) (p. 2)

Although we agree with Dourish’s point that algorithms and programs are not the same entities,

one is still left with identifying the defining properties of algorithms. Table 1 presents the conceptual

definitions and key attributes of algorithms from a review of the literature. It is apparent from this table

that there are a few core attributes of algorithms and that these have been somewhat stable over time. For

example, virtually all of the definitions contain references to a procedure or step-by-step instructions.

Many also explicitly discuss the need for an input into that process and the need for an output for that

process. It is interesting to note that the attributes can be applied to a variety of tasks that many would not

consider algorithmic. Baking a cake is a straightforward example. A cake is baked using a set of inputs

(i.e., flour, eggs, sugar, etc.), a procedure for combining inputs and baking, and an output (the cake). For

the present study, we are interested in how humans and algorithms interact in decision making. This

naturally leads us to consider the definition of algorithms through a technology or computer science lens.

9

Table 1:

Summary of Prior Algorithm Conceptualizations

Author(s) Conceptualization Attributes

Cormen (2009) "...an algorithm is any well-defined computational procedure

that takes some value, or set of values, as input and produces

some value, or set of values, as output." (p. 1)

Procedure; Inputs;

Outputs

Cormen (2013) "...a set of steps to accomplish a task." "a computer algorithm

is a set of steps to accomplish a task that is described

precisely enough that a computer can run it." (p. 1)

Precise; Steps; Task

Accomplishment

Dourish (2016) "In computer science terms, an algorithm is an abstract,

formalized description of a computational procedure." (p. 3)

Computational

procedure

Gillespie

(2014)

"Algorithms need not be software: in the broadest sense, they

are encoded procedures for transforming input data into a

desired output, based on specified calculations. The

procedures name both a problem and the steps by which it

should be solved." (p. 1)

Procedure; Inputs;

Outputs

Goffey (2008) "Algorithms do things, and their syntax embodies a command

structure to enable this to happen." (p. 17)

Structure

Harris and

Ross (2006)

"an algorithm is a set of well-defined steps required to

accomplish some task." (p. 1)

Well-defined steps;

Task

Accomplishment

Kitchin (2017) "sets of defined steps structured to process instructions/data

to produce an output." (p. 16)

Defined steps; Input

Instructions; Output

Kowalski

(1979)

"An algorithm can be regarded as consisting of a logic

component, which specifies the knowledge to be used in

solving problems, and a control component, which

determines the problem-solving strategies by means of which

that knowledge is used." (p. 424)

Input Knowledge;

Output; Control

Steps

Lewis and

Papadimitriou

(1978)

"a precisely stated procedure or set of instructions that can be

applied in the same way to all instances of a problem." (p. 96)

Precise procedure;

Problem Solving

Mundra and

Dwivedi (2013)

"An algorithm is the step-by-step solution to a certain

problem." (p. 1)

Steps; Problem

Solving

Stephens

(2013)

"An algorithm is a recipe for performing a certain task." (p.

3)

Task

Accomplishment;

Recipe

10

The input is perhaps the most complicated part of the definition to understand, as we can consider

many scenarios where there are zero (0) or more inputs. Some might consider zero to be the lack of an

input, thereby making inputs unnecessary as part of a definition of an algorithm. However, that is not the

case, as long as the procedure has some means of using this information. A simple example showing a

computational procedure that expects as an input some value between 1-20, but also incorporates a NULL

(Missing) input may help:

IF input value is NULL

print("No Number!")

ELSE IF input value < 10

print ("Lower than 10")

ELSE

print ("10 or Higher")

ENDIF

Given the attributes from Table 1. we can ascertain that an algorithm must have some means of

acquiring input, performing some computations or testing of logical conditions on that input, selecting

various actions that it may undertake, and then providing an output. This sequencing of steps requires that

algorithms be clearly defined and have a formal procedure or a "command structure," as Goffey (2008)

calls it. Cormen (2009) says, "We can also view an algorithm as a tool for solving a well-specified

computational problem. The statement of the problem specifies in general terms the desired input/output

relationship" (p. 5).

For this review, it will be sufficient to build upon these attributes, leaning heavily of the

definition and clarity provided by Cormen (2009), and define an algorithm in the following manner:

An algorithm is any well-defined computational procedure that takes some values as inputs and

produces some values as outputs. The procedure identifies both a problem and the steps by which

it should be solved. An example of an algorithm may be a program that takes historical sales data

as input and predicts future sales as output. Another example may be an algorithm that takes as

inputs a user’s past search behavior on a website and predicts what they should be shown on the

screen (see Figure 1.)

11

Figure 1 What Are Algorithms? Adapted from Cormen (2009)

Many times when we refer to algorithms, we are referring to a procedure related to cognitive

tasks or a prediction machine (Agrawal et al., 2018). The cognitive applications and uses for algorithms

create the possibility that these will conflict with human decision-making. This potential for conflict

illustrates a critical difference between algorithms and why they are adopted or avoided versus other

technologies.

The areas of technology, programming, and computer solutions have been an active area of

research for some time. What makes algorithms interesting now, and how do they differ from the

technologies researched in the past? Perhaps one of the most significant differences is that more and more

computers can act independently of human control. This independence and the burgeoning agency is in

contrast to a program procedurally carrying out the will of human operators (Hoc, 2000). The introduction

of autonomous activity that can be viewed by humans as a program acting as an autonomous agent

(Diakopoulos, 2014; Hoc, 2000) adds additional complexity to Human-Computer Interaction (HCI) or

cooperation. Autonomy and agency also introduce a conflict with human-based judgment and decision-

making (Dietvorst et al., 2015; Logg et al., 2018; Prahl and Van Swol, 2017). Although the discounting of

algorithms has a long academic history, their prevalence in our lives makes this a going concern.

Literature Review

In the following section, I review the literature on the Technology Acceptance Model (TAM),

technology diffusion, and the role that trust plays in the acceptance process. This review provides an

overview of some of the issues that need to be considered in developing a definition of algorithm

adoption, and distinguishing it from other, related constructs.

12

Technology Acceptance Models

The information systems field has done a significant amount of research on technology

acceptance models (Marangunić and Granić, 2015; Venkatesh, Davis, and Morris, 2007). Integrating

some of the findings from research on these models provides creators or users of algorithms with

potentially valuable insights if improving acceptance is the goal. Although technology and algorithms are

not synonymous, the factors that influence users’ acceptance of technology will undoubtedly have some

overlap with the factors that influence users’ adoption of algorithms.

In one of the most influential papers in the information systems literature, Davis (1989)

introduced the Technology Acceptance Model (TAM). This systematic study of acceptance revealed why

firms are not able to fully capture the value from the technology due to barriers in acceptance. TAM

introduced two constructs to explain a user's intention to use a given technology -- perceived usefulness

(PU) and perceived ease of use (PEU). Davis (1989, p. 320) defines perceived usefulness (PU) as "the

degree to which a person believes that using a particular system would enhance his or her job

performance." The second factor, perceived ease of use (PEU), was defined as "the degree to which a

person believes using a certain system would be free of effort." (Davis, 1989, p 320). Both factors were

identified as major contributing factors related to a person’s intention to use technology. In subsequent

research (Szajna, 1996; Venkatesh and Davis, 2000), there has continued to be strong support for these

factors. In particular, perceived usefulness has been shown to be the most significant predictor in

influencing technology acceptance. Perhaps to improve adoption, algorithm designers can focus on

designing useful algorithms that improve measurable outcomes? One challenge to this approach is the

nature of many of today's algorithms, which often underperform upon their introduction and then improve

over time.

Venkatesh and Davis (2000) extended the original TAM model by including factors of social

influence such as subjective norms and image, and cognitive processes such as results demonstrability

(see Figure 2). This study validated and replicated some of the original TAM findings across four

organizations longitudinally. TAM2 explained a significant portion of the variance in intention to use

13

(60%). One of the most important additions to the original TAM model was that subjective norms have a

direct effect on the intention to use and an indirect effect through perceived usefulness. Importantly,

another finding was that a compliance-based approach to encouraging adoption is less effective over time

than a campaign of social influence when it comes to changing user's perceptions of perceived usefulness.

Finally, this study provided practical advice to implementers to focus on demonstrating the comparative

effectiveness of new technologies to increase adoption.

Figure 2 TAM2 Venkatesh and Davis (2000)

Technology Diffusion

"Diffusion is the process by which an innovation is communicated through certain channels over

time among the members of a social system." Rogers (2010, p. 5).

If algorithm adoption can at least partially be explained by the adoption of other technologies

through the technology acceptance model (Davis, 1989), what other factors can explain how algorithms

are accepted up by users? One exciting avenue in the literature is technology diffusion, which provides a

lens through which to view adoption by firms and individuals. Rogers (2010) explains that diffusion is

about innovation and communication over time within communities. Although individual adopters may

evaluate a potential innovation by trying to understand its usefulness and ease of use (Davis, 1989), they

14

first must become aware of the innovation. Organizations are similar, in that the innovation must be

introduced by, or to, the organization. Rogers (2010) differentiates between passive and active "knowing"

about an innovation. Some potential adopters hear or learn about the innovation passively, and others seek

it out. This stage of knowledge is further broken down into awareness, information acquisition on using

the knowledge, and an understanding of the underlying principles of the innovation. How might these

different stages be related to algorithm adoption?

The awareness stage is critical, because as Rogers (2010) notes, it is at this stage where adopters

may seek out additional information on the algorithm. The next stage in this model is persuasion. In the

context given by Rogers (2010), this is where a unit of decision-making, the individual or the firm, forms

an opinion of the innovation. This is not by necessity a favorable opinion. This stage seems most related

to TAM (Davis, 1989), as it is at this point in TAM that adopters are making decisions on whether or not

the technology is useful in some way.

The confirmation stage involves a decision-maker seeking reinforcement of a decision already

made. Rogers (2010) discusses anecdotal cases of innovation discontinuance after adoption. Although

Rogers (2010) discusses some of the discontinuance drivers, a more thorough model is presented in

Bhattacherjee (2001). In this research, a model of technology continuance was developed to understand

the factors that lead users to abandon or retain a particular technology. Interestingly, almost all research

based on TAM has shown that Perceived Usefulness (PU) is the most significant factor in the acceptance

of technology. As previously discussed, this suggests that any measures of algorithm adoption should

include measures of the usefulness of the algorithm. As can be seen from Figure 3, confirmation of the

performance of the technology has an indirect effect on information systems (IS) continuance through its

effect on perceived usefulness and satisfaction (Bhattacherjee, 2001). Confirmation is a cognitive belief,

based on an expectation of a given outcome and realized by the user through use of a given technology.

15

Figure 3 Technology Continuance Bhattacherjee (2001)

Trust and Algorithms

Algorithm Avoidance

Of course, another factor that may influence be an individual's willingness to adopt an algorithm

is his/her openness to receiving innovations and fairly evaluating them; a concept McKnight et al. (1998)

calls "disposition trust." Other scholars have written extensively on the desire of decision-makers to be

consistent in their decision-making, and ensuring future decisions are consistent with their prior beliefs

(Moore and Small, 2007; Tversky and Kahneman, 1975; Yaniv and Kleinberger, 2000).

In many cases, the technology tested in the various versions of TAM, TAM2, and technology

diffusion models is similar to the algorithm-driven technology we are interested in. One of the primary

differences is that algorithms can change and adapt, while the implementation and interface stay the same.

This means that inputs and outputs can evolve in a non-transparent way. This inaccessibility can cause

conflicts with a user's decision-making and judgment. One way users may react to these changes is by

changing their trust in an algorithm. Trust has been shown in multiple studies to be an essential factor in

both human-human and human-automation literature. Before moving on, it is perhaps helpful to review

the concept of trust through the literature (see Table 2) and develop a common understanding of its

relevant attributes.

16

When most technologies or algorithms historically relied on straightforward "If this, then that"

steps, the user's accessibility to how the machine thought was straightforward. Now algorithms often have

complex inputs, inaccessible procedures, and probabilistic outputs, opening them up to significantly

different interpretations of effectiveness and trustworthiness. Lee and See (2004) discuss how trust in

humans differs from trust in automation. The primary difference is that automation lacks "intentionality"

and is not part of the social exchange. As noted by Lee & See (2004):

There is symmetry to interpersonal trust, in which the trustor and trustee are each aware of the

other’s behavior, intents, and trust (Deutsch, 1960). How one is perceived by the other influences

behavior. There is no such symmetry in the trust between people and machines (p. 66)

Lee and See (2004) explore the effect of trust in human-computer interaction. Their study sets out

clear steps firms or individuals can take to increase trust in automation, including setting an appropriate

expectation for performance and training those interacting with the automation on how and when the

automation can be relied upon.

The attributes of trust discussed above can be found in various fields on the adoption of advice

from other humans, automation, or algorithms. The trust attributes most relevant for algorithms in the

present study are found in Muir (1987): expectations, competency, and responsibility. In the human judge

advisor literature, an area primarily concerned with how humans interact with other humans in giving and

receiving advice, multiple studies have been conducted to understand the implications of violating some

aspect of trust (Sniezek and Van Swol, 2001). This research has had significant implications for human-

automation and algorithm trust research. In the forecasting literature, Sanders and Manrodt (2003) discuss

the impact of complicated solutions on decision-makers, finding that complicated solutions are relied

upon less. This may not be so different from the findings of TAM (Davis, 1989), which suggest that

Perceived Ease of Use (PEU) is a critical factor in technology acceptance.

17

Table 2:

Summary of Trust Conceptualizations

Author(s) Conceptualization of Trust Key Attributes

Deutsch (1973) "The confidence that one will find what is desired from another,

rather than what is feared" (p. 148)

Confidence,

expectation

Rempel, Holmes,

and Zanna (1985)

"a generalized expectation related to the subjective probability an

individual assigns to the occurrence of some set of future events"

(p. 63)

Expectation

Cook and Wall

(1980)

(Trust) "refers, in the main, to the extent to which one is willing

to ascribe good intentions to and have confidence in the words

and actions of other people." (p. 39)

Good intentions.

confidence

Mayer, Davis,

and Schoorman

(1995)

“The willingness of a party to be vulnerable to the actions of

another party based on the expectation that the other will perform

a particular action important to the trustor, irrespective of the

ability to monitor or control that party.” (p. 712)

Willingness to be

vulnerable, based

on an expectation

Mayer et al.

(1995)

"Three characteristics of a trustee appear often in the literature:

ability, benevolence, and integrity. As a set, these three appear to

explain a major portion of trustworthiness." (p. 712)

Ability;

Benevolence;

Integrity

Lee and See

(2004)

"The attitude that an agent will help achieve an individual’s goals

in a situation characterized by uncertainty and vulnerability." (p.

54)

Uncertainty;

Vulnerability

Bhattacharya,

Devinney, and

Pillutla (1998)

"Trust reflects an aspect of predictability-that is, it is an

expectancy." (p. 461)

Predictability

Bhattacharya et

al. (1998)

"Trust cannot exist in an environment of certainty; if it did, it

would do so trivially. Therefore, trust exists in an uncertain and

risky environment." (p. 461)

Uncertainty

Castelfranchi and

Falcone (2000)

"The word "trust" is ambiguous: it denotes both the simple

trustor’s evaluation of trustee before relying on it (we will call

this "core trust"), the same plus the decision of relying on trustee

(we will call this part of the complex mental state of trust

"reliance"), and the action of trusting, depending upon trustee."

(p. 3)

Reliability,

depending on the

trustee

Muir (1987) "Trust (T) is the expectation (E), held by a member (i) of a

system, of persistence (P) of the natural (n) and moral social (m)

orders, and of technically competent performance (TCP), and of

fiduciary responsibility (FR), from a member (j) of the system,

and is related to, but not necessarily isomorphic with, objective

measures of these qualities." (p. 531)

Expectation;

Competence;

Responsibility

18

The results of five studies by Dietvorst et al. (2015) show that errors in the output of an algorithm

make people less confident, and therefore less likely to adopt the algorithm's advice. The study suggests a

certain amount of intolerance for algorithmic error, as participants chose the algorithm that erred less

frequently, even when it outperformed a human forecaster. Similarly, Prahl and Van Swol (2017) tested

participants' reactions to the consistency of the errors of an algorithm. They found that inconsistent errors

have an adverse effect on adoption. This type of response may be related to a perfection schema

(Dzindolet, Pierce, Beck, & Dawe, 2002). The perfection schema is the theory that humans have an

expectation that technology will perform flawlessly. In a series of experiments, Merritt, Unnerstall, Lee,

and Huber (2015) found that some individuals think of automation as "all or nothing” and are less likely

to forgive automation when it errs.

Despite these robust results, recent research from Logg et al. (2018) showed that at least some of

the algorithm aversion found in prior studies was due to algorithmic advice conflicting with an

individual's assessments. Soll and Mannes (2011) discuss the effect of conflict with the self and the

overweighting of one's own opinions. Overall, Logg et al. (2018) found that decision-makers do use

algorithms over human advisors and that this use improves overall decision-making.

What role does an algorithm’s accessibility or interpretability play? Poursabzi-Sangdeh,

Goldstein, Hofman, Vaughan, and Wallach (2018) designed a study to understand these factors' effects on

an individual's adoption of an algorithm. Participants took part in forecasting tasks, estimating the price of

homes. In some cases, the participants saw an algorithm with an accessible model, the coefficients for

rooms, bathrooms, square footage, etc. In the alternate case, participants only saw a "black box" model

that came up with the final home price estimate. Counter-intuitively, they found that the accessibility of

the algorithms did not impact a user's level of trust in the algorithm, nor did the ability of participants to

spot or fix algorithm errors change. These findings should cause us to think critically about how to

measure algorithm interpretability and to build a model that allows us to understand when and how it acts

in the overall schema of adoption. Algorithm interpretability may not lead directly to adoption, but it may

interact with other decision-maker factors such as domain knowledge or experience.

19

In recent research, Castelo, Bos, and Lehmann (2019) conducted several large scale experiments

to understand the role the nature of the task had on acceptance. They found that algorithms were trusted

and therefore relied on less(more) for tasks that were perceived to be more subjective(objective).

Interestingly, they also found that the trust perception users had of the tasks could be changed. A users

perception of the trustworthiness of an algorithm could be changed by providing examples of algorithms

performing well on subjective tasks. Overall their findings suggest that the effectiveness of an algorithm

is a more important factor of algorithm acceptance than an individual’s discomfort with an algorithm

performing some task.

In the human-human literature, there is at least some weighing of interpersonal factors into advice

acceptance. In much of the human/algorithm literature to date, researchers are comparing the adoption of

advice acceptance between humans and algorithms. However, many of the most powerful algorithms

decision makers use do not compete with expert opinion. The automation trust literature is perhaps most

analogous to human-algorithm trust.

Machine and People Learning with Trust

If algorithms are often better than the humans they are replacing, why are they often ignored? It

could be that some experiences are just "better" delivered by a human (Agrawal et al., 2018). Yeomans,

Shah, Mullainathan, and Kleinberg (2017) conducted an interesting study on computer vs. human

generated recommendations. The research found that subjects considered computer-generated jokes

funnier than those generated by people. However, subjects were less likely to rely on recommendations

when they knew that a computer vs. a human had generated the jokes. The researchers conducted a

follow-up study, in which rich explanations of how the joke recommender algorithm worked were given

to test participants, and a sparse recommendation was given to a control group. The results were

illustrative; the group that received a detailed explanation of the workings of the algorithm rated the

algorithm as easier to understand (M = 0.09, 95% CI = [0.01, 0.18]) vs the low information group (M = -

0.09, 95% CI = [-0.18, 0.00]), t(984) = 2.93, P = .003). This research is important in the context of how

algorithms are built and deployed. The findings on subjective vs. objective acceptance of algorithms in

20

Castelo, Bos, and Lehmann (2019) would tend to support this idea, that there are just some domains and

contexts in which the advice of an algorithm is not accepted.

Another way algorithms might be considered less trustworthy than a human advisor is when

considering the interpretably of a given algorithm (Poursabzi-Sangdeh et al., 2018.) A significant effort

has recently been made to make algorithms more interpretable (Ribeiro, Singh, & Guestrin, 2016). These

solutions however try to solve an algorithmic problem in interpretability, with more algorithms. Despite

some of these efforts, the field of machine learning is still trying to improve the robustness of these

methods as recent tests of these methods have shown a lack of consistency in the outputs (Alvarez-Melis

and Jaakkola, 2018). This reduces their usefulness in the short term because a lack of consistency in

outputs can lead to lower adoption by the general public.

Given the amount of literature discussing the relative performance benefits of algorithms vs.

humans on many tasks, and the fact that the disparity in advice taking from machines is still a relevant

topic (Meehl, 1954; Dietvorst et al., 2015), it seems unlikely that there is a simple solution to the problem

of inadequate advice adoption. There have been efforts made to remedy some of the gaps. One innovative

solution is to make the process of accepting algorithmic advice more like that of accepting advice from a

human provider. Fridman, Ding, Jenik, and Reimer (2017) detail one such solution -- train more than one

algorithm to perform a task. If the algorithms agree, provide the user with some level of confidence that

these two "arguing machines" agree. When they disagree, explain or display the results in an interpretable

way, and allow the human to arbitrate. Competing dialogues such as this are much more similar to how

humans operate today; they often receive differing views and make a decision based on these differing

views. The impact on algorithm acceptance, especially in areas where the algorithm is taking some level

of control is potentially tremendous, Fridman (2018) calls this "shared autonomy."

When machines learn from their input-algorithm-output path, this is referred to as "Supervised

Learning" (Friedman, Hastie, and Tibshirani, 2001). It works through a series of feedback loops that

reinforce "correct" outcomes. This looping and re-measuring are why many machine learning algorithms

need so much data. Learning through feedback loops is not much different than the way organizations

21

learn (Rosenberg, 1982) by doing things and measuring the outcomes. Nor is it different than the way

most of us learn lessons throughout our lives. Perhaps then one way forward is for algorithm designers to

start small, and then gradually increase the level of algorithm autonomy. This stair-stepping of algorithm

autonomy is already happening, albeit likely due to technological restrictions versus design. Most of us

will have driven in a car that self-parks or has an emergency (algorithm-generated) stopping feature

before we own a car that drives itself.

This learning by doing also brings up a serious issue. If most of our learning as humans comes

from experience, how will we learn when our environment no longer gives us this experience? A chilling

example given by Agrawal et al. (2018) is the case of Air France Flight 447 that crashed in the Atlantic in

2009. The autopilot disengaged during a severe storm, and the junior pilot was unable to manage the

emergency. The senior pilot woke up and was not able to react fast enough to save the plane or the

passengers. Although the junior pilot had several thousand hours of flying, it had earned primarily on

autopilot. The de-skilling of pilots in this way means that the pilots will get worse as algorithms get better

at handling emergencies. Contrast the Air France Flight 447 result with the US Airways Flight 1549 that

landed on the Hudson River.

One way of looking at this might be that, for 42 years, I've been making small regular deposits in

this bank of experience: education and training, and on January 15, the balance was sufficient so

that I could make a very large withdrawal. - US Airways Capt. Chesley "Sully" Sullenberger.

("Capt. Sully Worried About Airline Industry." 2009).

Thinking critically about how to train people to accept an algorithm, and when to use an

algorithm, is not always as straightforward as it seems. Technology has not progressed to the point where

an algorithm would have successfully landed US Airway 1549. Ongoing de-skilling of pilots might mean

that in the future, there may not be a Captain Sully onboard.

Concept Development

This primary goal of this dissertation is to clarify the definition of the algorithm adoption

construct and its sub-dimensions, and to explain how this construct differs from other related constructs

such as TAM. The work follows the recommendations of Podsakoff et al. (2016) (see Figure 5) to create

22

concept definitions and refine those definitions through a review of existing literature and interviews with

subject matter experts.

Figure 4 Stages for developing conceptual definitions (Podsakoff et al., 2016)

In the first stage of the construct definition process, we reviewed the literature and dictionary

definitions "algorithm," “adoption,” and "trust." Based on this literature, our preliminary definition of

algorithm adoption is as follows:

Algorithm Adoption is the act of choosing to use an algorithm's output for some purpose by

individuals or firms. Looking back at our definition of an algorithm, Input(s)→Algorithm→

Output(s), an adopter will take the outputs of the algorithm and use these outputs to perform some

type of action. Like the conceptualization of this in @rogers2010, an adopter is a unit of analysis

and may be an individual or a firm. Of course, it is possible for algorithms to be compiled

together, each taking outputs from the preceding algorithm. In this case, algorithm adoption is

determined programmatically with either deterministic or probabilistic steps. However, the

current study explicitly refers to algorithm adoption by firms and individuals and not by other

algorithms.

It is important when defining a construct to define both what it is and what it is not (MacKenzie,

2003; MacKenzie, Podsakoff, and Podsakoff, 2011). The confusion between acceptance and adoption is

one such case and requires clarification. Acceptance refers to the act or feeling of receiving willingly,

approving of, or having a favorable response to something (Merriam-Webster.com, 2020). The definition

of adoption has key phrases such as: "to take up and practice or use," "to accept formally and put into

effect" (Merriam-Webster.com, 2020). Given these straightforward definitions, it is easy to see that the

act of accepting and adopting an algorithm are temporally related. An example may illustrate this. We

may accept the fact that driverless cars are coming, and we may accept that this technology (will be

someday) better than most, if not all, human drivers. However, this level of acceptance does not mean we

will adopt, or actively use, driverless cars. To be sure, acceptance is a strong predictor or of actual use

(Davis, 1989), but it is not equivalent to it. Much of the work on algorithms, and the related concepts of

automation and cognitive algorithms, simply assumes that solutions that perform better than a human at

23

the same or similar tasks will be adopted. Most of these works ignore the genuine possibility that workers

confronted with new solutions will fail to adopt them appropriately.

So, the next step in our process was to identify the key dimensions of the algorithm adoption

construct that help explain whether it is adopted or not. In order to refine the construct definitions,

interviews were conducted with professionals who are currently working or have worked extensively,

with algorithms in decision-making contexts.

Participants

The primary source of information for this stage is interviews with current and former

professional business people. Inclusion criteria for the participants were a minimum of 1 year of

experience in a role that required or requires decision-making. Some of the subjects were known to the

researcher. Others were introduced to the researcher as part of the interview vetting process. The roles of

the participants did not necessarily involve creating algorithms used to make decisions. However, each of

the interviewee's current or previous roles would have had multiple occasions to consider using an

algorithm to assist in making decisions. The thirty interviewees (7 females, 23 males) work experience

spans 1955 and 2019, with a median work experience of twenty (20) years. Participants also discussed

their work experience in terms of using algorithms in their decision making, the median experience with

algorithms was just over ten (10) years. All participants had a college education, and seventeen (17) held

post-bachelor’s degrees. Seventeen were, or had previously been, employed at large publicly traded

company, and nine were or had previously been employed in a consulting company, and one was

employed at a non-profit. The business sectors represented by the sample include hospitality, finance and

banking, retail, manufacturing, consulting and technology among others. See Table 3 for an overview of

the interviewees.

The use of subject-matter experts as a source of information in the conceptual development stage

of construct development is well supported by the literature (Podsakoff et al., 2016). Second, because this

group extends across multiple generations, the research covers the period when algorithmic decision-

making first arose in organizations, which helps identify differences in the cohort based on age, or work

24

experience. Third, interviews allow for a depth of concept development not available using other methods

(Venkatesh, Brown, & Bala, 2013).

Table 3

Interview Participants

Interview Number

Years of

Experience

Role

1. Sergio 20 Chief Research Officer, Consulting Company

2. Craig 50 Owner/Principle Boutique Market Research. Former VP Fortune 100

3. Judy 34 Director Finance, Fortune 500 company

4. Ted 50 SVP (Various roles) Fortune 500 company

5. David 12 SVP Mid-size Analytics Consulting Co

6. Chuck 32 Chief Brand Officer- Large Fast Casual Restaurant Chain

7. William 26 CEO/President Large Restaurant Brand

8. Jake 2 Big 4 Consulting Co

9. Gordon 15 Health Care Analytics

10. Hollis 3 Data Science Analyst- Large Finance/credit card company

11. Mary 8 Director, Real Estate Services firm

12. Susan 5 Analyst, Banking Industry

13. Jennifer 2 Analyst, Energy/Oil Company

14. Alice 1 VP Analytics/Insights, Banking

15. Paul 30 Former Sales Executive

16. Darryl 2 Analyst, Home goods/products company

17. Edward 26 Finance and Investing

18. Anthony 52 Founder - Market research company

19. Charlotte 10 Director Analysis, Non-Profit

20. Travis 19 Private Consulting

21. Earl 25 Former VP- Fortune 500

22. Noah 7 Data Scientist- Tech 'Unicorn' Company

23. Matthew 30 Former CEO, Board Member (start-up and established brands)

24. Jerry 13 Director Analytics- Travel Industry

25. Emanuel 30 VP Hospitality/Hotel Industry

26. Luke 24 Senior Leader- Equipment Manufacturing

27. Cameron 8 Developer (Multiple Tech and Tech-enabled companies)

28. Miles 20 COO Analytics Company

29. Jim 20 CEO, Analytics Company

30. Jacque 24 CFO/President Analytics Company

Note: Pseudonyms have been used to identify participants, rather than their real names.

Procedure

Qualitative data was collected through semi-structured interviews related to the participant's

experiences using algorithms to assist with decision-making. The structure and definitions of an algorithm

25

and its components were used to generate ten questions (See Appendix 1 for the detailed question list).

The interviews were conducted in various business offices in the United States, England, Singapore,

Tokyo, and France and via recorded calls. All interviews were recorded and transcribed. The transcripts

and descriptive information (dates, masked contact information, gender, age, experience, and professional

role(s) of the interviewee) were input into a data table maintained by the researcher. The transcripts were

also loaded into MAXQDA software (VERBI Software, 2020) to facilitate coding. Descriptive statistics

were calculated in the R programming language (R Core Team, 2020)

Each interview was individually analyzed to identify themes or "codes" that reflect the concepts

being studied. Consistent with Saldaña’s (2009) statement that "...the qualitative analytic process is

cyclical rather than linear" (p. 58), the coding process was split into several cycles. The first coding cycle

for each transcript was done using structural codes (Guest and MacQueen, 2008). Structural codes serve

to identify areas of the transcript that are related to topics of semi-structured interviews. This approach

allowed the researcher to query the data and review similar areas of the text across all participants to aid

in the next cycles of coding.

The next phase involved In-Vivo coding. This coding technique uses the participant's own words

as codes (Corbin, Strauss, & Strauss, 2015). This technique is most closely related to and discussed as a

part of the grounded theory technique. Although the present research takes a more conceptual and broader

view, it was expected that using the participant's own words (in vivo) would provide another basis for

reviewing, restructuring, and identifying the factors related to algorithm adoption.

Finally, the last and most detailed phase was concept and pattern coding. Saldaña (2009) argues

that concept coding is appropriate for studies on theory development. This phase involved iterating over

the interviews and coding each multiple times as/until themes emerged. The codes were generated first by

individually coding interviews, and later by using the structural codes from the first coding cycle to look

for similarities and differences across the participants. Finally, as themes emerged, these pattern codes

were grouped and used to organize the prior codes.

26

At the end of each interview, participants were asked to rate their willingness to adopt an

algorithm in a verbal survey consisting of 10 items on a 5-point Likert scale ranging from 1 (not an

important part of the decision) to 5 (an important part of the decision). Although the data is ordinal, to aid

in interpreting participant's relative importance scores for each attribute, Means (𝑥‾) and the Standard

Deviations (𝜎) were derived for each factor.

Results and Discussion

The purpose of this stage of research is to help identify, define, and develop factors that lead to

algorithm adoption by decision-makers. After several rounds of coding, four key dimensions of the

algorithm adoption construct were identified: (1) Input trust (referenced by 83% of the interviewees) (2)

output trust (70%), (3) understandability (57%) and algorithm province (60%). These themes, the first-

order categories that they are comprised of, and examples of the quotes made by the interviews are

summarized in Table 4, and Table 5 provides a summary of the percentage of participants that identified

each of these themes in their interview. Finally, Table 6 provides a summary of the descriptive statistics

from the survey asking the participants in the study to rate the importance of various factors to their

willingness to adopt and use an algorithm to help you make decisions in their jobs. In the section that

follows, we define these sub-dimensions of the algorithm adoption construct.

Table 4

Summary of Attributes of Algorithm Adoption Identified by Interview Participants

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

Input Trust

(Referenced by

83%)

The right

number of

inputs

"…more variables are usually better… to a point. You get to a

point where you're no longer adding value; you are just adding

noise. " (Charlotte)

"…some people feel like more is always better. So, the more

information we can shovel in here, the closer we'll get; when I

think that actually can produce a worse result. So, I actually feel

that sometimes decision-makers overdo it when it comes to what to

include." (Charlotte)

“…part of it was when we have to make decisions on our own on

whether or not to include as many variables as possible or just

27

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

make executive decisions on what doesn't seem to make sense,

based on our experience." (David)

"really about focusing on the two, sometimes three, most salient

inputs that drove that output." (Earl)

Understanding

Inputs

"I have to understand the data myself. With that, I think that's... I

don't want to say necessarily feature engineering, but maybe along

those lines, right? Why are we using these particular columns,

fields, whatever? Are they directly necessary? Are they a proxy for

something else? Do we have other, better data?" (Gordon)

“feature engineering is still the art of data science. The reason we

say that is because in order for you to actually derive the maximum

benefit, you have to think about what contributes the most to your

models.” (Noah)

“qualitative research will identify if there's something that doesn’t

fit…we never replace, we always add something else in so we can

maintain that consistency.” (Anthony)

Input

Accuracy

“What I've often found is that the data they're using to put in them

has been flawed and so what is produced is the outcome of the

algorithm's calculation is inaccurate and inappropriate even though

the basic formula may be fine.” (Craig)

"but if I have garbage in or if my assumption's incorrect, I mean,

my perceived quality of the data of the output is completely

inaccurate or incorrect." (Jake)

"data is messy" (Jake)

“I think that in terms of the outputs, the part that we want to

understand better before using them is whether the data that is used

to create those outputs is accurate.” (Sergio)

“Some of our clients have issues giving us actual or let's call it

accurate costs…. in many cases what we need to do is we need to

use proxies … How far we move away from the ideal? I guess it is

going to determine whether we can trust the final output” (Sergio)

“you're feeding it bad information, poor information, which then

the algorithm's just going to continue to get worse and worse. And

then, it's really not trusted” (Emanuel)

- Inputs make

sense

"…a lot of it is what makes sense to me. So, if I think about the

business problem I'm trying to solve for, have we captured the

28

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

population I think needs to be considered to derive the expected

output?" (Judy)

"Well, I think the most important thing would be whether the inputs

would make business sense or not in the specific use case that I'm

trying to solve" (Hollis)

“…it depends who is consuming the results. For example, this

product that I am currently working on I mentioned, there are a lot

of stakeholders involved. So, in this instance, everything needs to

make intuitive sense” (Alice)

“There are some things that are very logical in terms of ‘these are

the inputs that make up x’ and some things that are not. You have

things that occasionally are, what's the right word? I guess more

subtle.” (Earl)

- Input

completeness,

sufficiency

"…do we have all the inputs that we want in a perfect world?"

(Sergio)

"I like to think about specific other data sets that may exist" (Craig)

“Some of the outputs are just trash... we just don't have the right

variables in our data sets” (Darryl)

“The more we try some of these models…the predictions just don't

come out where we want them to be, the more I realize how

important it is to have the right set of attributes in your data, and

how hard it can be to collect that kind of information.” (Darryl)

“Actually, a lot of the work that I do is designing or creating the

algorithms that will allow the end client or the end-user to make

better decisions. So, defining how they're making decisions today,

what inputs might be missing to make a better decision…is the core

of what I do” (Travis)

“You have to have a good working knowledge of the data you have

and be objective about what you don't have.” (Matthew)

“The data might be insufficient to draw a reasonable conclusion.”

(Luke)

Output Trust

- (Referenced by

70%)

“Makes Sense” "It's going to be what does your gut say? Does it make sense to

you? Is it at all what you expected the result to be?" (Judy)

“is it consistent with common sense, or not?” (Craig)

29

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

"I think it was more of a confirmation to them that some of the

outputs made sense and that it was the right thing to do to act upon

those things." (David)

“Does it make sense? Can you build a story from it because it

should make sense and line up with other research that's going on?"

(William)

"it's sort of the trust but verify, meaning is it reasonable to expect

the output that I'm getting? I should, I think, walk in with an

expectation of what those results are going to look like." (Gordon)

"typically with any type of algorithm, you have some type of

expected result. If it's in a range of what you would expect, then I

guess that would be an answer of ‘yes, this is accurate.’" (Susan)

“if you don't question if the way in which you constructed the

algorithm and the results that you're getting make sense and are

right for that situation, then you may get into a situation that okay,

you get the results of the algorithm and you apply it without

thinking about what may be wrong with it and what the

implications are.” (Sergio)

“…I think that's where that sense check come from just looking at

it and getting an idea of does this pass the sniff test for me?"

(Emanuel)

Error “… the downsides of using an algorithm depends on the negative

impact when the algorithm fails, right? How big of a knee jerk

reaction do business leaders have when the algorithm makes an

error”? (Jake)

Replicability “…play devil's advocate, because if you're telling me it's supposed

to do something along the way, if I'm duplicating that in those same

steps and it doesn't match what you're giving me, I want to be able

to tie that out.” (Gordon)

Understandability

(Referenced by

57%)

Explainability “It got us enough accuracy within limits of our data, so we'd know

we'd be doing basically the right decision without basically make it

so opaque that no one would trust it because they didn't understand

what was going on” (Luke)

"The last thing is that what I like personally, although I find this is

typically not done in business by people. I want someone to explain

to me the logic of the algorithm. Am I going to be as proficient as

they are, including probably in writing the algorithm? At this point

in my career, probably not. But I'm still proficient enough that I can

30

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

understand whether the logic flow makes sense on the formula for

the algorithm. I want to understand enough about it to believe that

there's not some fundamental logic flaw in how they did the

calculation." (Craig)

“…saying is it something that we're going to be able to use in the

conversation with an executive, and explain, and justify versus

something that we would have a hard time explaining any kind of

relationship between the variable and the potential outcome."

(David)

"Is it something that we can explain?" (David)

"…when you're trying to explain it to people in the business maybe

they don't understand some of the terminologies that you're using,

or the steps that you use. I've seen that happen a lot here

specifically a lot of times people will ask me or other members of

my team to simplify it down to make things explainable to the

business."

(Hollis)

"At the end of the day what ended up happening was the whole

model got translated into something which was simple, decision-

based, which made sense for, you know like it was explainable to

everybody" (Hollis)

"I need to know how it works for it to be believable for me. And so

I think usefulness and explainability are critical" (Paul)

"…explainability, interpretability of outputs are huge for us."

(Darryl)

"One of the things that I feel is the hardest area in this domain is

the ability to explain predictions. The real push here is to be able to

explain the model downstream so that individuals who are being

impacted by the decision can actually go back and look at it and see

why that decision was made. And ultimately, you know, be able to

like change those, changed some of those parameters and make

sure that they can take some actions on that on that prediction."

(Noah)

"And explainability and acceptance by the user base. I'd say that's

absolutely critical and often overlooked with not just the technical

side of things, but how well it will be absorbed." (Luke)

"I think one of the things that we're always facing is how good a

model is predicatively, versus the ability to explain." (Miles)

31

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

Interpretability

"… if I'm going to pass on it, I better really understand how that

can be interpreted and applied." (Craig)

"Interpretable, it's totally possible to have a look at the output of a

boosted tree, but it's possible that I can interpret the results just as

easily. At least with the output. But I don't know that that translates

into something that I can tell a story around." (Miles)

"The models are very good at identifying those and making them

parsimonious. So that happens during the model phase. But

obviously, you also want to make sure that this is aligned with

intuition you're not building the models at a pure black box."

(Noah)

"I thought it was a really important evolution that we needed to

have on the sales team to be able to talk to the clients in their terms

and get rid of that black box effect." (Earl)

"…the black box algorithm has marginal value. To me, anyway,

given my background in understanding of technology and things

like that, I need to know how it works for it to be believable for

me." (Paul)

"So, kind of taking a very transparent box approach as opposed to

black box. So, having the client understand how the algorithm

works and the benefit and the compromises of the different

elements." (Travis)

"Yeah, our predictive model to predict balances over time, I think

the impact of the business is very strong if the model is not very,

very precise, and I think most people acknowledge that so I think

we have much more flexibility there to be a little bit more black

box." (Alice)

"Well, I feel like a lot of algorithms that we've used, there's a whole

quote, unquote ‘black box’, but we have a few of them that we're

actually able to kind of able to peel it back a little bit and

understand what's going on under the hood, so I guess

interpretability is big for us with our business partners. We go to a

lot of business partners who are savvy people, who don't

necessarily have data science or analytics degrees, but they get it.

They know their stuff." (Darryl)

"…black boxes, you don't really understand what is going in, unless

you build the whole thing from scratch." (Hollis)

"But I can drive a car without knowing how to build one, right?

And I think a lot of models and algorithms are really going that

way." (Miles)

32

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

Algorithm

Provenance

(Referenced by

63%)

Data

Provenance

(data quality) “If I can trace back where the data came…if I was the

business analyst, I'd probably go back to the source and see what

was happening that day and if it was accurate.” (Susan)

"Data is messy, and they come from siloed databases"

(Jake)

Process

credibility

“To me a lot of the results that I get like make sense in a way that

like not everybody else might be able to understand them, because

like when you're trying to explain it to people in the business

maybe they don't understand some of the terminologies that you're

using, or like the steps that you use.” (Hollis)

“(we used)…the very simple linear model and that allowed us,

when we got a new decision maker into the room, we could draw

this on the board really quickly they got it. Regression gave us a

straight line. It all made sense.” (Luke)

Author

Credibility

“We're still very much getting the credibility and we're trying to get

them to trust us to do our analytics.” (Alice)

“At the senior level, they don't necessarily know the algorithms.

What they really have to do is they have to have faith in the people

that actually do those.” (Craig)

“But most don't do that in business. They just take it ... They buy

into the expertise of who's sitting in front of them, and that

expertise they basically…they take blindly frankly.” (Craig)

"It is not just what the algorithm shows; it is who is providing it

within these structures. So often the evaluation, validation, and

acceptance of the algorithm's output is really anchored more in who

brings it, than what it actually is in the sense of its intellectual

integrity, or necessarily to some degree the quality of the output. I

think that is a fascinating part of this process."

(Craig)

“It's like I put my trust in who is building it.” (Matthew)

"Do you trust the person that's designing it, writing it, and

reviewing it?" (Matthew)

“If it was somebody I was working with for the first time and it

didn't pass my crap detector, I'd probably push back a little bit and

would require a higher level of convincing to understand that it is

correct and there's not a mistake in the algorithm.” (Matthew)

33

Second-Order

Themes

First-Order

Categories

Exemplary Quotations

“I think it's critical to not just get the results of the algorithms, but

to give your confidence explicitly to the users. So that they know

how much you trust your answer... I think that's absolutely vital to,

to ensure buy-in so they know how much you trust it so they're

going to get it more or less way to appropriately, as they trust you.”

(Luke)

"The advantage is it is not subjective. It's objective. If you run it

twice, you're going to get the same answer instead of just running it

and what they're feeling like that day. So that takes a lot of the

guesswork about what the motivation is behind the person making

the recommendation."

(Luke)

"You don't need explicit understanding. I would say that when I use

an algorithm or a library or something else that somebody else

wrote, I assume that they're much smarter than me and they know

what they're doing."

(Cameron)

"I found that for the most part, the people who develop algorithms

tend to do it out of necessity for their own project or their own

situation, not for ... And maybe sometimes for altruistic reasons.

Not necessarily for, "Oh, I'm in this arena and I want everybody

else to be worse at it than me."

(Cameron)

Table 5

Summary of Themes Reported by Participant

Key Attribute

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 #

% of

Interviews

Input Trust ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 25 83%

Output Trust ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 21 70%

Understandability ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 17 57%

Algorithm

Provenance ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 19 63%

Note. A block (◼) means the attribute was present in the respective interview. % of Interviews is the % of

all 30 interviews that discussed a given attribute.

34

Table 6

Descriptive Statistics for Verbal Algorithm Adoption Factor Survey.

Factor M SD

Quality of information obtained from the algorithm 4.43 0.77

Perceived usefulness to your job 4.33 0.84

Trust in the outputs of the algorithm 4.33 1.03

The interpretability of the algorithm 4.30 0.88

Past experience with using the specific algorithm 4.13 0.82

The nature of the algorithm’s inputs 4.13 0.97

Perceived ease of use 3.90 0.99

Past experiences with using other algorithms 3.70 0.95

Subjective norms of the organization 3.07 1.44

Knowing who the algorithm was developed by 2.93 1.44

Note. M and SD represent means and standard deviations, respectively. N=30.

Input trust. Recall that trust between machines and people has the key attributes of expectation,

competence, and responsibility (Muir, 1987). Additional factors such as ability, integrity, vulnerability

(Mayer et al., 1995), and uncertainty (Lee and See, 2004), can all be found in the literature on trust

between people and within organizations. In explaining why they accepted or rejected the use of an

algorithm, most interviewees (83%) mentioned some form of input trust. Although the specific

dimensions of input trust were varied, most participants (56% of all interviews) brought up issues of a

data input’s 'completeness' or 'sufficiency.' Sufficiency did not just relate to an algorithm's ability to make

better predictions, or the desire for more data, but also to what could be explained and what “made

sense.” There were in many examples a reference to the right number of variables, and sometimes a

tradeoff between an algorithm's performance needs and the needs of the decision-maker:

Keeping it to a somewhat reasonable number of variables to keep it actionable because

sometimes, I think it's that balance between going overboard from the number of variables to

supposedly get a better model or what have you and keeping it actionable. -David

Other interviewees (23%) discussed the accuracy of the inputs as being a relevant dimension in

accepting/rejecting the use of an algorithm. This relatively low percentage was somewhat surprising since

a common saying in many fields related to data analysis is "garbage in garbage out." As Jake explained,

"if I have garbage in... the quality of the data of the output is completely inaccurate or incorrect." Craig

differentiated input quality and the quality of the algorithm:

35

What I've often found is that the data they're using to put in them has been flawed and so what is

produced in the outcome of the algorithm's calculation is inaccurate and inappropriate even

though the basic formula may be fine. -Craig

Based on the participants’ comments, we defined Input Trust as the degree of trust a user has in

the data input into an algorithm by a decision-maker. It is based on the user’s confidence in the accuracy,

reliability, and timeliness of the data input into the algorithm. Input trust does not include factors related

to the algorithm’s creator or architect. Nor does it include an evaluation of the algorithm’s outputs.

Despite the high incidence of comments regarding inputs during the interview process, when

asked to score the importance of the nature of inputs on a decision to adopt an algorithms' decision,

participants scored this item in the middle of the ten items being tested (𝑥‾= 4.13, 𝜎=.97). However, it is

worth noting that the question on the survey was related to the nature of an algorithm’s inputs, not the

accuracy or quality of the inputs that most participants mentioned. Thus, it is possible that the reason for

the relatively moderate rating of the importance of this factor in terms of an algorithm adoption may have

had to do with the fact that the question did not accurately reflect participants’ trust in input quality, and

that if it had reflected this variable better, it would have been rated higher in terms of its importance.

Output trust. Most interviewees (70%) mentioned some form of output trust during the course of

the interview. This finding is generally consistent with the results of the two survey questions that asked

about the outputs of an algorithm. For example, when asked how important the quality of the outputs is to

an adoption decision, participants scored this the highest of any factor (𝑥‾= 4.43, 𝜎=.77). Similarly, when

asked how important "trust" in an algorithm's outputs is, participants ranked this tied for the second most

important factor (𝑥‾= 4.33, 𝜎=1.03). The amount of trust that users have in the output of the algorithm is

one of the main factors in whether they adopt the algorithm or not.

By far, the most frequently referenced theme to emerge from the research was the notion that an

output (result) had to "make sense" to the user. If it did not make sense, it was subject to additional

scrutiny in some cases, or it was simply rejected. The “makes sense” filter was an especially critical factor

for those interviewees whose job it was to explain the output of the algorithm to someone else. Some

interviewees expressed unease at noting that their “gut feelings” were an essential part of their decision-

36

making criteria. For example, one interviewee said, "Unfortunately, it's the worst answer I could give you,

it's gut." For the most part, participants discussed using this as a validation of some sort:

Basically, to me, an algorithm is here to validate mathematically my gut feel, right? My gut feel

tells me this right? Now I have all these inputs that are coming from those algorithms. Do those

two make sense? -Jacques

Because typically with any type of algorithm, you do have some type of expected result. If it's in

a range of what you would expect, then I guess that would be an answer of yes, this is accurate -

Susan

Taken together, these comments led us to define Output Trust as the degree of trust a user has in

the outputs of an algorithm because the outputs are considered reasonable and credible by the users, the

output meets the user’s expectations, and the algorithm is believed to have performed its task competently

and correctly. Output trust does not involve an evaluation of an algorithm creator’s competence, but

rather a user’s belief that the steps the algorithm performed we’re executed correctly.

Similar to Mayer et al. (1995), the consequences of a violation of output trust are partially

determined by factors such as the nature of the task being performed, the perception of risk to a decision-

maker, and the alternatives to using the algorithm. It is logical to extend these factors to include domain

knowledge. A simple example: A driver follows Google(TM) Maps (an algorithm-based solution), and an

alternative route is identified that purports to save 3-minutes on a 30-minute drive. The risk (total drive

time) is low, the alternatives are all available to the decision-maker, and the domain knowledge of the

decision-maker will become the most significant factor. Does this 3-minute alternative route "make

sense" given what the driver knows of this route? However, the decision is likely to change when the time

saved begins to increase. Indeed, as discussed in the next section, 47% of respondents discussed prior

knowledge and “domain knowledge” as an important consideration in deciding whether or not to accept

an algorithm’s output.

Understandability. A large number of interviewees (57%) mentioned a preference for using

models that are easy to understand when considering whether to adopt an algorithm into their decision

making. An analysis of the interviewees reveals two sub-dimensions of this desire for understandability:

Interpretability (33%), and explainability (43%). It is worth noting that participants had a hard time

37

defining precisely what these two terms represent; indeed, none of the participants were able to offer a

concise definition. They are not alone. Academic interest in both interpretability and explainability to

improve model adoption is growing, but clear concept definitions remain elusive (Gilpin et al., 2019).

Despite the lack of clear definitions of these concepts from participants, when asked how important

interpretability was in their decision to adopt an algorithm, it was the fourth-highest factor on the survey

(𝑥‾= 4.3, 𝜎=.88).

One obvious way to make models more interpretable is to keep them very simple. As has long

been discussed in the academic literature, increased adoption of even simple models can improve decision

making (Dawes, 1979). The field of computer science and statistics has a number of models, such as low

dimensional decision trees and simple linear models that may be highly interpretable (Friedman, Hastie,

and Tibshirani, 2001). Unfortunately, these simple models provide interpretability at the expense of

predictive accuracy. More sophisticated models, from highly dimensional decision trees to deep learning

models, may have vastly superior predictive accuracy but are, for the most part, "black boxes." This

tradeoff was apparent in the results of the interviews. Interviewees had strong but not always consistent

opinions of non-interpretable models. In general, interviewees rejected the use of these types of models in

their decision making:

…the black box algorithm has marginal value. To me, anyway, given my background in

understanding of technology and things like that, I need to know how it works for it to be

believable for me. -Paul

I better really understand how that can be interpreted. -Craig

Some interviewees were more forgiving of the black box, acknowledging that a better predictive

model is more important than a model's interpretability:

I think the impact of the business is very strong if the model is not very, very precise, and I think

most people acknowledge that so I think we have much more flexibility there to be a little bit

more black box. –Alice

These diverging opinions highlight an aspect of understandability that is perhaps important to

discuss here. It is relevant to consider the domain expertise of the prospective users of the algorithm. It

seems logical that users who have a significant amount of algorithm expertise, as users or creators, would

38

evaluate a model's interpretability very differently. The domain knowledge factor is perhaps one of the

reasons that experiments testing the direct effect of interpretability on algorithm acceptance do not always

show that interpretability is a significant factor (Poursabzi-Sangdeh et al., 2018.)

A closely related factor to interpretability is explainability. In computer science, explainability is

generally understood to be: can the model sufficiently explain its processes and the reasons for the output

and provide insights into the main drivers of those outputs. Data analyst D understands algorithm A by

means of the algorithm’s own explanation E (adapted from De Regt, Leonelli, and Eigner, 2009, p 23) is

one such way of clarifying this view of explainability. For clarity, this definition is the computer or

algorithm explaining itself. If explainability is the algorithm's ability to explain itself, a reasonable

question to ask here might be "explain to whom?" Like with interpretability, it seems reasonable that the

explanation one user finds sufficient, is not at all suitable for someone with a different background.

Based on these comments, Understandability is defined as the extent to which a user feels that the

algorithm is not only interpretable but is also able to explain its steps/thinking processes. To be

interpretable, an algorithm must be readily understood by the user. To be explainable, an algorithm must

be able to make clear for a human user its processes, inputs and outputs. Understandability does not

involve an evaluation of the appropriateness or fit of a given algorithm to a problem, but rather just the

user’s ability to comprehend what the algorithm is doing.

Algorithm provenance. Although the participants in the study rated the importance of knowing

the source of the algorithm relatively low in terms of their willingness to accept its output (𝑥‾= 2.93,

𝜎=1.44), an interesting pattern emerged during the coding of the interviews in relation to an algorithm, its

component pieces, and its provenance or history. Provenance is an active area of research in databases

and information retrieval (Buneman, Khanna, and Tan, 2001) and is discussed in relationship to

computational tasks by Freire, Koop, Santos, and Silva (2020). Prat and Madnick (2008) discuss

provenance as "believability" and outline several dimensions related to believability, including the

trustworthiness of the source, the reasonableness of the data, and the temporality of the data. Through the

interview process, themes related to provenance as it pertains to algorithm adoption were generated and

39

are presented in Table 4. Overall, 63% of participants identified some form of provenance as being a

factor in adoption. The difference between the ratings on the survey and the interviews may relate to the

wording of the question on the survey. The survey question asked participants whether knowing the

author of an algorithm was an important factor. It is, of course, possible to know an author and have

differing assessments of the credibility of that source. In contrast to knowing, interviewees spoke on the

credibility of an author, algorithm history, and credibility of the process through which the algorithm was

generated.

Another aspect of algorithm provenance is the credibility of the algorithms author, creator, or the

firm that produced it. No consensus was found among interview participants concerning the credibility

and motivation of the authors of algorithms that they had experience with. Some participants noted that an

algorithm’s author deliberately lied, as Ted states, “We still had people that cooked the books.” Others

did not feel like the credibility or capability of an author of an algorithm is particularly pertinent, or felt

that it is unlikely that the authors had motivations that were not aligned with their own.

I've found that for the most part, the people who develop algorithms tend to do it out of necessity

for their own project or their own situation, and maybe sometimes for altruistic reasons. Not

necessarily for, 'Oh, I'm in this arena and I want everybody else to be worse at it than me.-

Cameron

The advantage is it is not subjective. It's objective. If you run it twice, you're going to get the

same answer instead of just running it and what they're feeling like that day. So that takes a lot of

the guesswork about what the motivation is behind the person making the recommendation. -Jake

Based on these comments, Algorithm Provenance is defined as the degree of trust a user has in an

algorithm because of a user’s assessment of the credibility of the advisor that developed the algorithm.

Algorithm Provenance is based on (a) the trustworthiness (or credibility) of an algorithm’s advisor

(person or firm), (b) the advisor’s ability to clearly explain the processes and steps by which input data

are transformed to create the output, and (c) the advisor’s ability to explain the algorithm’s output.

Algorithm provenance does not refer to a computer or algorithm's ability to explain its processes, but

rather it is specific to the human advisor or agent. Algorithm Provenance does not refer to the correctness

of the algorithm’s outputs or the data used as inputs in the algorithm.

40

The aspect of provenance most prevalent in the literature relates to the sources and history of the

data being used in the algorithm. Scholars have long been concerned with ensuring that the source of data,

or its combination and manipulation, is accurate (Buneman et al., 2001). This sentiment was echoed by

the participants, several of whom spoke about making efforts to ensure the source was correct. As Gordon

said: "We want to look at what are the main sources of data across a number of different business units.

With that, we want to look at what we call sources of truth." In the present study, this concept likely has

overlap with "Input Trust" and is measured as part of that factor.

Refining the Concept of Algorithm Adoption

Based on the interviews and review of the existing literature, this paper proposes that Algorithm

Adoption is a higher-order construct with four first-order sub-dimensions (Input Trust, Output Trust,

Understandability and Algorithm Provenance) that reflect the higher-order construct (see Figure 6).

.

Figure 5. Proposed Model for the Dimensions of Algorithm Adoption.

Relationship to Other Constructs

Podsakoff et al. (2016, p. 186) have noted that when defining a construct, it is critical to

distinguish it from other, related, constructs.

It is also important to differentiate the focal concept from other, related concepts. Indeed, the very

act of defining/labeling something (saying what it is) requires distinguishing it from other things

(saying what it is not). The benefit of doing this is that it (a) helps to distinguish the attributes that

define the focal concept from the attributes that define other, related concepts; (b) diminishes the

possibility of concept proliferation; and (c) identifies the concepts that could be used in empirical

tests of the measures of the focal concept’s discriminant validity (p. 186).

41

So, in the section that follows, I discuss the similarities and differences between Algorithm

Adoption the Technology Acceptance Model (Davis, 1989) and the trust in automation model introduced

by Lee and See (2004). The similarities and differences in these models that are discussed below are

summarized in Table 7.

Comparison to TAM

The three major models based on the technology acceptance model (TAM, TAM, and UTAUT)

are presented in Figure 7, panels A, B, and C, respectively. These models share some limited traits with

the algorithm adoption model (AAM). Conceptually at least, the perceived ease of use (PEU) in TAM and

understandability (in AAM) share a common antecedent in the form of complexity. As Thompson,

Higgins, and Howell (1994) discuss, an increased level of complexity makes it harder to use a given

technology. It stands to reason that complexity also impacts the two sub-dimensions of understandability,

interpretability, and explainability, in much the same way. A more complex system is, by its nature, more

difficult to interpret and explain.

Figure 7. Evolution of TAM models. Adjusted graphic from Holden, R. J., & Karsh, B. T. (2010).

42

Table 7

Comparisons of TAM, TAM2, UTAUT, and Automation Trust Concepts with AAM. Construct Definition Similarities Differences

TAM,

TAM2,

UTAUT

(Davis 1989;

Venkatesh

and Davis

2000;

Venkatesh

2003

TAM ≈ AAM. Both models seek to explain the factors related

to behavioral intention. TAM (acceptance) and AAM

(Adoption).

TAM ≠ AAM. TAM assumes that technology acceptance is

primarily driven by a technology’s perceived ease of use

(PEU) and perceived usefulness (PE). TAM is a broad model

and thought to cover most cases of technology. AAM, on the

other hand, assumes that an individual’s trust in an algorithm's

inputs and outputs are a primary factor in adoption. The AAM

model does not seek to explain technology acceptance overall

but treats algorithms as unique.

TAM technological ≠ AAM Trust/social..As described by

Gefin (2003), “Trust is a social antecedent. Perceived ease of

use and perceived usefulness are technological antecedents.”

Perceived Ease of Use

(PEU)

“The degree to which a

person believes that using a

system would be free of

effort” - Davis (1989, p.

320)

PEU (TAM) ≈ Understandability (AAM). These constructs

share a common antecedent in the form of complexity. As

complexity increases, PEU would decrease, as would an

algorithms interpretability and explainability.

PEU (TAM) ≈ Understandability (AAM). Although not

explicated in the model, PEU must be related to an

individual’s abilities and capabilities with that technology. In

AAM, the Understanding concept shares this trait. Different

individuals will perceive different levels of interpretability and

explainability based on their familiarity with that algorithm or

algorithms like it.

PEU ≠ Understandability. An algorithm may be very easy

to use, and yet not be understandable.

Perceived Usefulness

(PU)

"the degree to which a

person believes that using a

particular system would

enhance his or her job

performance" -Davis (1989,

p. 320)

PU ≠ Output Trust. A review of the items in PEU shows that

the primary focus is on increasing productivity or the

simplicity of using technology. These are technological

factors, not trust-evaluation factors.

43

Automation

Trust

(Various)

Automation Trust ≈ AAM. Trust precedes reliance in the

majority of automation trust conceptual models and AAM.

Automation Trust ≠ AAM. AAM explicitly separates the

trust a user may have with the inputs, outputs, or provenance

of an algorithm. Existing models describe trusting beliefs in a

more general way.

Dispositional/Propensity to

Trust

"The level of trust that exists

based on one’s past

interactions with the

machine." (Merrit, 2008, p.

197)

Propensity to trust ≈ Input/output trust. Propensity to trust

should be an antecedent to input and output trust.

Perfection Automation

Schema (PAS)

The expectation that

technology should behave

perfectly.

(Dzindolet et al. 2002)

Perfection Automation Schema (PAS) ≈ Output trust.

Output trust contains an element of PAS in that output trust

requires that “algorithm is believed to have performed its task

competently and correctly.”

Perfection Automation Schema ≠ Output trust. PAS is used

in the context of human decision making vs. automation

assisted decision making. The study considered an “all or

nothing” adoption or rejection of automation advice for each

decision. The factors of output trust are “degrees.”

Performance concept

Refers to the automation’s

reliability, predictability, and

ability.

(Lee & See, 2004)

Automation Trust ≈ Output trust. The trust in automation

model includes the concept of performance, which These

roughly correlate with output trust in terms of the user's belief

that the task was performed by the algorithm competently and

correctly.

Automation Trust ≠ Output trust. Output trust does not

require that an algorithm performs in a predictable way, but

that the output is reasonable and credible, and meets

expectations.

Faith

An aspect of belief beyond

the evidence that is available

(Rempel et al. 1985)

Faith ≈ Output Trust. Output trust includes conditions that

could be related to faith, such as "the algorithm is believed to

have performed its task competently and correctly."

Faith ≠ Algorithm Provenance. Trusting can be informed

through experience, evidence, and belief. Algorithm

provenance relates to an assessment of the credibility of the

algorithms creator. It does not rely on “blind faith” in that

creator.

≈ Indicates a concept is compared to or related to its counterpart. ≠ Indicates a concept is not equal to its counterpart and will be contrasted.

44

Although not explicated in the TAM model, PEU must also consider an individual’s abilities. The

ease of use for a given technology must be related to an individual’s abilities and capabilities with that

technology. In AAM, the understanding concept shares this trait. Different individuals will perceive

different levels of interpretability and explainability based on their familiarity with that algorithm or

algorithms like it.

Putting aside the trust dimensions in AAM, which are not present in TAM, these technology

acceptance models different from the AAM in two ways. First, TAM assumes that PEU and PE primarily

drive technology acceptance. TAM is a general model and designed to cover a large number of cases of

technology. AAM, on the other hand, assumes that an individual’s trust in an algorithm’s inputs and

outputs, the algorithm’s provenance, and the algorithm’s understandability are the primary factors in

adoption. The AAM model does not seek to explain technology acceptance overall but treats algorithms

as unique.

Second, although perceived usefulness would seem to share at least some overlap with the AAM

dimension of output trust, a closer examination of the items (Table 8) used to measure perceived

usefulness (Venlatesh, 2003) makes the differences clear. As indicated in this table, items such as "using

the system would make it easier to do my job" and "Using the system increases my productivity" focus on

technological aspects that relate to improving one’s productivity/job performance, and are not consistent

with the themes of output trust, such as the credibility of the outputs or meeting a user’s expectation.

Table 8

Items used to measure Perceived Usefulness and Performance Expectancy Attribute Items

Perceived

Usefulness

(Davis 1989)

1. Using the system in my job would enable me to accomplish tasks

more quickly

2. Using the system would improve my job performance.

3. Using the system in my job would increase my productivity.

4. Using the system would enhance my effectiveness on the job.

5. Using the system would make it easier to do my job.

6. I would find the system useful in my job

Performance

Expectancy (TAM)

(Venkatesh 2003)

1. I would find the system useful in my job.

2. Using the system enables me to accomplish tasks more quickly.

3. Using the system increases my productivity.

4. If I use the system, I will increase my chances of getting a raise

Table adapted from Venkatesh (2003).

45

Comparison to Automation - Trust

One of the most influential works on trust in automation is the conceptual model of trust and

reliance developed by Lee and See (2004). This automation trust (AT) model (see Figure 8) is built on the

work of Fishbein and Ajzen (1975), which posits that an attitude is an affective evaluation of beliefs,

leading to an intention to adopt automation. In addition, the AT model proposes a feedback loop, where

interaction with the automation and updating of information continues to form beliefs, leading to an

evolution of trust. This model is similar to the AAM framework in that trust plays a key role in both

models. For example, trust precedes reliance in the Automation-Trust model, and it precedes adoption in

the AAM framework. The trust in the AT model includes the concept of performance, which refers to the

automation’s reliability, predictability, and ability. These roughly correlate with output trust, in terms of

the user’s belief that the task was performed by the algorithm competently and correctly. In addition, the

dispositional propensity to trust included in the AT Models would be expected to influence both input

trust and output trust in the AAM framework.

Figure 8. Lee and See (2004) p. 68. Conceptual model of trust and reliance

46

Despite these similarities, the treatment of trust in the Lee and See (2004) AT framework differs

from AAM in several ways. First, AAM explicitly separates the trust a user may have with the inputs,

outputs, or provenance of an algorithm, whereas the AT framework describes trusting beliefs in a more

general way. This makes sense when the goal of the model is to consider automation broadly, but does not

fit into the way algorithms are either created or used.

Second, although both the AAM and the AT framework share the goal of improving user

information to support decision making (AAM through the concept of understanding, and AT through the

concept of information display), there are also some crucial differences. For example, AAM does not

require or discuss the format of the display as an important factor, whereas AT considers it important. An

example of this might be a simple calculator. Understandability in AAM requires explainability and

interpretability. If a user enters 2 + 2 on the calculator, and it returns 4, then a user can both interpret and

explain that calculator algorithm. It is inarguable that the 4 is a type of display, of course. This display

assists users by allowing them to see that the outputs are reasonable and credible, and meets the user’s

expectations, an important component of output trust. But extending this idea, the calculator itself, the

plastic, buttons, and screen, is the wrapper around the algorithm. This wrapper is also the display. That

display, the quality of the buttons, the feel of the calculator may have a significant impact on a user’s

decision to use that particular calculator. But, the technology tool adoption decision is separate from the

adoption or acceptance of the algorithm. For the purposes of the AAM model, the function inside the

calculator is what is essential.

The perfection automation schema (PAS) introduced by Dzindolet et al. (2002) has already

received some attention in this study. An experiment by Dietvorst et al. (2015) showed that when

algorithms err, they are relied on less. The concept of PAS is related to the concept within output trust of

"belief" that a task was performed competently by an algorithm. PAS also extends somewhat further than

the AAM model, in considering whether an automation acted in some predictable way. AAM has no

correlate for a predicted outcome, but this is at least somewhat similar to the output "meeting the user’s

expectations." Prediction and expectations do not have precisely the same meaning, but neither can they

be entirely separated. A particular outcome may be both predicted to occur and expected to occur. A

47

prediction implies a probable outcome of some future event. An expectation, on the other hand, implies a

more certain outcome. If a more certain outcome fails to occur, it is likely this triggers a more negative

emotional reaction from a user than a violation of a user’s prediction.

Although not specifically a model for automation trust, Rempel, Holmes, and Zanna (1985) have

been a foundation of later work and is relevant to the concept of algorithm adoption. Rempel et al. (1985)

describe one aspect of trust, faith, as an aspect of belief beyond the evidence that is available. The act of

trusting itself is informed through experience, evidence, and belief. The belief/fair framework differs from

the conceptualization of algorithm provenance in AAM, which relates to an assessment of the credibility

of the algorithm’s creator. It does not rely on “blind faith” in that creator. It is hard to argue that the

concept of faith has no relationship to output trust, however. Specifically, output trust incorporates within

its definition, "the algorithm is believed to have performed its task competently and correctly." This belief

could indeed be generated by a process such as faith.

Algorithms, from this paper’s perspective, will live within a wide variety of contexts. Those

contexts are not fully explored here, but it is, of course, essential to develop an understanding of a user’s

adoption of a given algorithm based on both the algorithmic component, technology usability,

accessibility, and display components. Table 8 provides a summary of the critical similarities and

differences between AAM, TAM, and the automation trust literature.

Antecedents of Algorithm Adoption

One of the goals of this research is to identify some of the key antecedents to algorithm adoption.

This was achieved through a review of the literature and an analysis of the interviews. The most

prominent antecedents in the literature come from the technology acceptance model. Performance

expectancy and effort expectancy could reasonably be considered as antecedents, as they are considered

before an algorithm is used.

As noted earlier, the literature on algorithm adoption is mostly experimental in nature, and very

little has been done to define or explore the construct outside of these experiments. However, these

experiments do highlight several potential antecedents. For example, studies on algorithm errors show

that within a study, participants rely on algorithms less after they err, or when the errors are not

48

predictable (Dietvorst et al., 2015; Prahl and Van Swol, 2017). The literature on trust suggests that

violating a user's trust by showing errors reduces the reliance on algorithms. The trust literature also has

several relevant antecedents to consider. Perhaps the most relevant, Dispositional Trust is defined by

Merritt and Ilgen (2008) as, "the level of trust that exists based on one’s past interactions with the

machine" (p. 197). Although the definition of dispositional trust explicitly discusses one's experience with

a particular machine, it is sensible to consider whether an individual's experience with all machines is a

relevant factor.

Interestingly, few antecedents to a decision maker’s adoption of an algorithm were identified in

the interviews. Even when asked directly about reasons for not adopting or using algorithms, very few

participants discussed errors in other algorithms being the cause of a change in their adoption intention for

all algorithms. However, as already noted, a large number of interviewees discussed rejecting the use of a

given algorithm when the outputs did not "make sense" to them.

Domain Expertise: Another important factor discussed by a large number of participants (47% of

interviews) was the role that domain knowledge or domain expertise plays in adopting decisions

generated by an algorithm. This was often mentioned in connection with or as an explanation for the

rejection of an algorithm when it did not "make sense."

If you use the algorithm, it would tell you something ridiculous...and it gets you ridiculous

results. So, in that case, we didn't use an algorithm, we used basically instincts, past experience,

interviews with salespeople -Jake

I think, for me, this is the balance between domain expertise and quantitative knowledge

experience. Whatever the word you want to use there. The domain expertise is that ... You know,

it's funny, because this is what I say to people is: How do you feel when you take that outlier and

put it into a sentence? –Miles

It is perhaps relevant here to recall the research of Kuncel et al. (2013), who found that a

mechanical combination of results outperformed human judgment. This is very much in alignment with

earlier research by Meehl (1954). Kuncel et al. (2013) state: "While recognizing that a strong preference

for expert judgment makes a complete change in practice unlikely" (p. 1070) and go on to list several

potential remedies for the case being investigated. It is interesting to consider how the preference of

experts to override and overrule, even in the presence of evidence that is contrary, can be moderated.

49

Not all of the discussions on domain expertise were about rejecting or adopting algorithms.

Participants spoke at length about things like asking the right questions, or understanding the output

within a business context:

You know your space very, very well and you know when something's good and when

something's not very good... I can say, you know what? There is value there. Nope, there is not

value there. -Edward

One of the most sought after skills for my team is having somebody that can merge the

quantitative aspect of what we do with the business environment -Sergio

The survey asked participants to rate their willingness to adopt an algorithm based on experience

with a particular algorithm (𝑥‾= 4.13, 𝜎=.82), and based on experience with all algorithms (𝑥‾= 3.70,

𝜎=0.95). Neither factor scored very high, and it is perhaps not surprising that experience with a particular

algorithm has a stronger relationship to adoption.

Benefits and Barriers to Algorithm Adoption

The benefits of adopting algorithms into decision making seem clear. A considerable amount of

literature has been published on the appropriate adoption of algorithmic advice. This was largely echoed

throughout the interview process, with 93% of subjects describing the benefits of using algorithms as

improving decision-making quality and efficiency. In particular, interviewees describe the ability of

algorithms to find and exploit new opportunities.

Algorithms give us the capacity to really take data, information, facts, and convert it into insights

that are actionable -Craig

The biggest (benefit) is the fact that it kind of removes your personal bias -Sergio

The benefit is it can do things faster, consistently, every single time the same way. That is hugely

powerful. Humans cannot do that. -Chuck

The biggest benefit is, I think it allows scientific decision-making, and it is quick. It's very, very

quick. -Darryl

The barriers to adopting algorithms also seem clear. Algorithms can be complex (40% of

interviewees) and can lack accessibility (43% of interviewees). These issues were most often discussed in

concert with comments about interpretability and explainability. Overall complexity was a driving force

50

in a lack of adoption in decision making downstream from the interviewees. No interviewees discussed

complexity as an impediment to their use of an algorithm.

Timeframe and execution capability are definitely factors for it not being implemented. -Jerry

…they just wouldn't believe it. It was too complicated -Judy

When asked about the biggest downside of using algorithms in decision making, two main

themes emerged. The first was a general concern that using an algorithm can lead decision-makers to

ignore other potentially more critical factors, or that a lack of a clear algorithmically derived answer

would limit any decision making at all.

I've seen leaders get hamstrung to the point they can't make a decision, because the data just isn't

clear enough for their level of comfort.- Matthew

The second downside to algorithm adoption was a sort of truth bias. For example, a number of

interviewees (30%) discussed seeing algorithms being adopted, even when they should not be. There has

been a substantial amount of attention on algorithm bias in the literature and the news, and the issue is one

that deserves more attention. It could be that some decision makers become resistant to adopting

algorithms into their decision making because of their inappropriate adoption elsewhere.

Sometimes algorithms can make people who may not have full understanding or a full grasp of

what we're trying to accomplish, the objective, make them feel smarter than they might be

because the algorithm is working so hard - Chuck

I think that sometimes people may get into this automatic situation of the algorithm is always

right…stop thinking about the problem - Sergio

These barriers are difficult problems to solve, as they invite discussion into not just the algorithm,

which by its nature is simply a code executing some command, to the psychological decision-making

traits, bias, ability and perhaps even willingness of the users to adapt and change to using new solutions.

Summary and Implications

Through an examination of the literature, and input from 30 practitioner interviews, this study

examined the nature of algorithm adoption and developed conceptual definitions for several proposed

sub-dimensions of the construct. An analysis of the interview content revealed that four common themes

51

adequately covered the majority of the rational decision makers gave when describing how they decided

to accept or reject the use of an algorithm:

• Input Trust: Mentioned by 83%

• Output Trust: Mentioned by 70%

• Algorithm Provenance: Mentioned by 63%

• Understandability: Mentioned by 57%

When discussing how and when they choose to adopt an algorithm into their decision making,

many interviewers used non-quantitative approaches like "makes sense" (mentioned by 60%) or the

variety and veracity of the inputs (mentioned by 57%). With only a few exceptions, very few described

techniques to quantitatively query or test the model prior to making a decision.

Understandability and its subdimensions of interpretability and explainability were also

frequently discussed factors. Improving model interpretability is an important topic at computer science

conventions and by computer science scholars. There has been significant improvement in building

interpretability frameworks for complex algorithms, such as those popularized by Ribeiro et al. (2016).

However, these advances in improving black box models do not seem to have reached practitioners yet.

Practitioners still prefer to use algorithms that are interpretable in and of themselves. This gap may have

significant consequences. As Lipton (2018) notes, the goals of increasing interpretability and avoiding the

use of black boxes may be directly at odds with the goal of improved predictive accuracy.

The short-term goal of building trust with doctors by developing transparent models might clash

with the longer-term goal of improving health care. -Lipton (2018, p. 21)

It seems clear from the interviews and academic work to date that a user's trust in an algorithm is

evaluative. In this conceptual scheme, users of an algorithm first evaluate the inputs, outputs, and

provenance of a given algorithm against factors that are important to them and base their trust in the

algorithm on this evaluation. As discussed in the examination of the literature, studies that attempt to test

the direct effect of these factors on algorithm aversion and adoption do not always have consistent or

intuitive results (Dietvorst, 2016; Logg, Minson, & Moore, 2018; Poursabzi-Sangdeh et al., 2018; Prahl &

52

Van Swol, 2017). Given a trust-evaluation scheme, this is perhaps not surprising. A more in-depth

examination of the factors that may mediate the trust-evaluation scheme users have with algorithms

would move the field forward substantially. The concept developed here of understandability may be one

such factor.

This study also examined the relationship between the proposed construct of algorithm adoption

and other similar constructs, such as those specified in the technology acceptance model (Davis, 1989)

and the literature on automation trust. There are some similarities between the proposed algorithm

adoption model and TAM. One similarity is the TAM model’s perceived ease of use concept and the

AAM model’s understandability concept sharing a common antecedent in the form of complexity. There

are some key differences as well. The TAM model primarily considers factors of acceptance to be

technological, while the AAM model considers trust-evaluation factors and understanding. A more

nuanced example of a difference is found between the TAM factor of perceived usefulness, and the AAM

concept of output trust. To be useful to a user, one may presume that the user would have to trust that

technology, but TAM makes no assertions regarding trust in a given technology. Output trust, on the other

hand, does not require that a given technology be useful in some way. It should be evident that any future

model of algorithm adoption carefully considers and integrates the concepts found in the technology

adoption literature.

When considering the similarities between the automation trust literature and the algorithm

adoption model, it is unavoidable to see the similarities across the different concepts. Various scholars

have discussed the relevance of factors such as propensity to trust (Merrit, 2008), the perfection

automation schema (Dzindolet et al. 2002), and the concept of automation performance (Lee & See,

2004). The contribution of AAM is a model explicitly targeting algorithm adoption and incorporating

many of the prior works into a single model.

Another goal of this study was to understand the antecedents of algorithm adoption. Surprisingly,

very few of the interview subjects discussed antecedents as it relates to their decision to accept or reject

an algorithm. The literature on trust in automation would seem to suggest that factors such as

dispositional trust should be significant (Merritt & Ilgen, 2008). In addition, 43% of interviewees

53

discussed the role that domain expertise plays into algorithm adoption decisions. This factor was most

often brought up in connection with other factors of trust, such as whether to trust the output of a given

model.

The benefits of adopting algorithms into decision making seem clear. Interviewees describe using

algorithms because these improve their decision making in terms of quality and efficiency. Many describe

algorithms as a crucial part of their decision-making process. The barriers to adopting algorithms were

also evident: Algorithms can often be complicated and lack accessibility. This complexity can lead to a

lack of understandability in an algorithm and diminishes a decision makers reliance on it.

Contribution

There is a long history in the academic literature on the importance of using models to assist in

the decision-making process. These algorithm advisors are often seen as making far better predictions and

decisions than their human counterparts. However, recent experiments (Castelo et al., 2019; Dietvorst et

al., 2015) highlight certain conditions under which this advice is underutilized. These concrete examples

of algorithm aversion and adoption help us understand what happens when various algorithm-generated

advice variables such as an algorithms subjectivity vs. objectivity are changed. However, they fail to

capture the concepts at a higher level, and so they do not adequately describe the domain of algorithm

adoption. Each of the experimental factors is like looking through a narrow lens and seeing only one part

of the answer.

The current research on the concept of algorithm adoption has been insufficient, with little to no

effort in understanding conceptually how human decision-makers take advice from algorithms. What

seems to be missing is a basic understanding and definition of the factors of algorithm adoption that will

serve to advance this construct of algorithm adoption as a theory. A gap this paper seeks to partially fill.

Clear definitions are useful in a number of ways.

a. Future research can start the process of developing content valid measures of the

Algorithm Adoption construct.

b. With clear definitions, researchers can test new relationships between the newly defined

algorithm adoption construct, and other variables such as TAM and trust factors.

54

For practitioners, the overwhelming support for the sub-dimensions can be a road map. First,

decision-makers can use this to understand their decision making processes further. Practitioners can

develop a deeper understanding of how factors such as their desire for an algorithm to ‘make sense’ can

lead them to under adopt algorithms. It is also useful to consider how an algorithm's provenance, such as

the credibility of the creator, may lead decision makers to adopt an algorithm that they should not adopt.

Second, for those involved in selling or distributing algorithmically generated advice, these findings

illustrate clear guidelines as to what decision-makers look for when considering the adoption of this

advice.

Conclusion

Making decisions with algorithms is growing in importance in the workplace and at home

(Agrawal et al., 2018). Existing literature in information systems and psychology have all developed

perspectives on adoption and use of technology in various forms. Literature from the Information Systems

(IS) field has shown us a strong relationship between the usability and usefulness of a technology and the

rate or amount of adoption (Davis, 1989; Venkatesh et al., 2003). But that research does not cover the

types of interactions that human decision-makers have with algorithms. The field of psychology is

undoubtedly developing the idea of trust and the perfection schema with algorithms, and finding that

when trust is violated adoption drops (Dietvorst, 2016; Prahl & Van Swol, 2017), but does not incorporate

the factors from relevant IS research. This study improves our understanding of the factors that lead to

adoption and builds a foundation for incorporating the disparate academic work to date.

Algorithms and their use in decision-making tend not to be discussed broadly. When they are

discussed, it is usually the adverse outcomes that are possible when delegating decision making to a

machine. Things like bias and the ways in which algorithms have erred tend to make better news stories

than the many ways algorithms can enhance and improve human decision making. Algorithms can be a

powerful decision-making aid, freeing people to do the critical tasks they do best and leaving the rest to

the machine. This is only possible if we develop a much deeper understanding and appreciation for how

we accept and incorporate algorithmic advice, confront the bias both in ourselves and in algorithms and

55

build a more algorithmic inclusive and aware decision-making paradigm. The management field is at the

forefront of the disruption that algorithms will create in the business world. By choosing to understand

and adapt to the changes, practitioners and scholars can develop a deeper understanding of how best to

ride this wave of disruption, instead of being drowned by it.

56

References

Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction Machines: The simple economics of artificial

intelligence. Harvard Business Press.

Ajzen, I. (1991). The Theory of Planned Behavior. Organizational Behavior and Human Decision

Processes, 179–211.

Alvarez-Melis, D., & Jaakkola, T. S. (2018). Towards Robust Interpretability with Self-Explaining Neural

Networks. arXiv Preprint arXiv:1806.07538.

Bhattacharya, R., Devinney, T. M., & Pillutla, M. M. (1998). A formal model of trust based on outcomes.

Academy of Management Review, 459–472.

Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation

model. MIS Quarterly, 351–370.

Buneman, P., Khanna, S., and Tan, W. C. (2001). Why and where: A characterization of data provenance.

In J. VanDenBussche and V. Vianu (Eds.), Database Theory - Icdt 2001, Proceedings (Vol.

1973, pp. 316–330). Berlin: Springer-Verlag Berlin.

Castelfranchi, C., & Falcone, R. (2000). Trust and control: A dialectic link. Applied Artificial Intelligence,

14(8), 799–823.

Castelo, N., Bos, M. W., & Lehmann, D. R. (2019). Task-Dependent Algorithm Aversion. Journal of

Marketing Research, 809–825.

Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation

criteria for definitional correspondence and definitional distinctiveness. Journal of Applied

Psychology, 1243–1265.

Cook, J., & Wall, T. (1980). New work attitude measures of trust, organizational commitment and

personal need non-fulfilment. Journal of Occupational Psychology, 39–52.

Corbin, J., Strauss, A. L., and Strauss, A. (2015). Basics of Qualitative Research. SAGE.

Cormen, T. H. (2009). Introduction to algorithms. MIT press.

Cormen, T. H. (2013). Algorithms unlocked. Cambridge, Massachusetts: The MIT Press, [2013].

57

Crano, W. D., Brewer, M. B., & Lac, A. (2014). Principles and methods of social research. Routledge.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information

technology. MIS Quarterly, 319–340.

Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American

Psychologist, 571.

Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 1668–1674.

Definition of ACCEPT. 2020. In Merriam-Webster.com. Retrieved June 5, 2020, from

https://www.merriam-webster.com/dictionary/accept.

Definition of ADOPTING. 2020. In Merriam-Webster.com. Retrieved June 5, 2020, from

https://www.merriam-webster.com/dictionary/ADOPTING.

Definition of UNDERSTAND. 2020. In Merriam-Webster.com. Retrieved June 20, 2020, from

https://www.merriam-webster.com/dictionary/UNDERSTAND.

De Regt, H. W., Leonelli, S., & Eigner, K. (2009). Scientific understanding: Philosophical perspectives.

University of Pittsburgh Pre.

Deutsch, M. (1973). The resolution of conflict. New Haven, CT: Yale.

Diakopoulos, N. (2014). Algorithmic Accountability Reporting: On the Investigation of Black Boxes.

Dietvorst, B. J. (2016). People Reject (Superior) Algorithms Because They Compare Them to Counter-

Normative Reference Points (SSRN Scholarly Paper No. ID 2881503). Rochester, NY: Social

Science Research Network.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid

algorithms after seeing them err. Journal of Experimental Psychology: General, 114.

Dourish, P. (2016). Algorithms and their others: Algorithmic culture in context. Big Data & Society,

2053951716665128.

Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002). The perceived utility of human and

automated aids in a visual detection task. Human Factors, 79–94.

February 10, Katie Couric CBS, 2009, “Capt. Sully Worried About Airline Industry.” Retrieved

June 21, 2020 (https://www.cbsnews.com/news/capt-sully-worried-about-airline-

58

industry/).

Field, A., Miles, J., & Field, Z. (2012). Discovering statistics using R. Sage publications.

Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behaviour: An introduction to theory and

research (Vol. 27).

Freire, J., Koop, D., Santos, E., & Silva, C. T. (2020). Provenance for Computational Tasks.

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). Springer

series in statistics New York, NY, USA:

Fridman, L. (2018). Human-Centered Autonomous Vehicle Systems: Principles of Effective Shared

Autonomy. arXiv:1810.01835 [Cs]. Retrieved from http://arxiv.org/abs/1810.01835

Fridman, L., Ding, L., Jenik, B., & Reimer, B. (2017). Arguing Machines: Human Supervision of Black

Box AI Systems That Make Life-Critical Decisions. arXiv:1710.04459 [Cs]. Retrieved from

http://arxiv.org/abs/1710.04459

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). Springer

series in statistics New York, NY, USA:

Gefen, David, Elena Karahanna, & Detmar W. Straub. "Trust and TAM in online shopping: An integrated

model." MIS quarterly 27.1 (2003): 51-90.

Gillespie, T. (2014). The relevance of algorithms. Media Technologies: Essays on Communication,

Materiality, and Society, 167.

Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2019). Explaining Explanations:

An Overview of Interpretability of Machine Learning. arXiv:1806.00069 [Cs, Stat]. Retrieved

from http://arxiv.org/abs/1806.00069

Goffey, A. (2008). Algorithm. In Software studies: A Lexicon, ed. Matthew Fuller, 15-20. Cambridge,

MA: MIT Press.

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical

prediction: A meta-analysis. Psychological Assessment, 19.

59

Guest, G., & MacQueen, K. M. (2008). Handbook for Team-based Qualitative Research. Rowman

Altamira.

Harris, S., & Ross, J. (2006). Beginning algorithms. Indianapolis, Ind. : Wiley, 2006.

Hoc, J.-M. (2000). From human Machine interaction to human Machine cooperation. Ergonomics, 833–

843.

Holden, R. J., & Karsh, B. T. (2010). The technology acceptance model: its past and its future in health

care. Journal of biomedical informatics, 159-172.

Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication &

Society, 14–29.

Kowalski, R. (1979). Algorithm= logic+ control. Communications of the ACM, 424–436.

Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical Versus Clinical Data

Combination in Selection and Admissions Decisions: A Meta-Analysis. Journal of Applied

Psychology, 1060–1072.

Lee, J. D., & See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human

Factors, 50–80.

Lewis, H. R., & Papadimitriou, C. H. (1978). The efficiency of algorithms. Scientific American, 96–109.

Lipton, Z. C. (2018). The mythos of model interpretability. Queue, 16(3), 31–57.

Logg, J., Minson, J., & Moore, D. A. (2018). Algorithm Appreciation: People Prefer Algorithmic To

Human Judgment (SSRN Scholarly Paper No. ID 2941774). Rochester, NY: Social Science

Research Network.

MacKenzie, S. B. (2003). The dangers of poor construct conceptualization. Journal of the Academy of

Marketing Science, 323–326.

MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation

procedures in MIS and behavioral research: Integrating new and existing techniques. MIS

Quarterly, 293–334.

60

Malone, T. W. (2018). How Human-Computer’Superminds’ Are Redefining the Future of Work. MIT

Sloan Management Review, 34–41.

Marangunić, N., & Granić, A. (2015). Technology acceptance model: A literature review from 1986 to

2013. Universal Access in the Information Society, 81–95.

Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust.

Academy of Management Review, 709–734.

McKnight, D. H., Choudhury, V., & Kacmar, C. (2002). Developing and validating trust measures for e-

commerce: An integrative typology. Information Systems Research, 334–359.

McKnight, D. H., Cummings, L. L., & Chervany, N. L. (1998). Initial Trust Formation in New

Organizational Relationships. Academy of Management Review, 473–490.

Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the

evidence.

Merritt, S. M., & Ilgen, D. R. (2008). Not all trust is created equal: Dispositional and history-based trust

in human-automation interactions. Human Factors, 194–210.

Merritt, S. M., Unnerstall, J. L., Lee, D., & Huber, K. (2015). Measuring Individual Differences in the

Perfect Automation Schema. Human Factors, 740–753.

Moore, D. A., & Small, D. A. (2007). Error and bias in comparative judgment: On being both better and

worse than we think we are. Journal of Personality and Social Psychology, 972.

Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of

adopting an information technology innovation. Information Systems Research, 192–222.

Muir, B. M. (1987). Trust between humans and machines, and the design of decision aids. International

Journal of Man-Machine Studies, 527–539.

Mundra, S. B., & Dwivedi, G. K. (2013). Design and Analysis of Algorithms. University Science Press.

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk.

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2016). Recommendations for creating better

concept definitions in the organizational, behavioral, and social sciences. Organizational

Research Methods, 19(2), 159–203.

61

Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018).

Manipulating and measuring model interpretability. arXiv Preprint arXiv:1802.07810.

Prahl, A., & Van Swol, L. (2017). Understanding algorithm aversion: When is advice from automation

discounted? Journal of Forecasting, 691–702.

Prat, N., & Madnick, S. (2008). Measuring data believability: A provenance approach. In Proceedings of

the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008) (pp. 393–

393). IEEE.

R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R

Foundation for Statistical Computing.

R Core Team. (2020). R: A language and environment for statistical computing. Vienna, Austria: R

Foundation for Statistical Computing.

Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985). Trust in close relationships. Journal of Personality

and Social Psychology, 95.

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Model-agnostic interpretability of machine learning.

arXiv Preprint arXiv:1606.05386.

Rogers, E. M. (2010). Diffusion of innovations. Simon and Schuster.

Rosenberg, N. (1982). Inside the black box: Technology and economics. Cambridge University Press.

Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage

Publications Ltd.

Sanders, N. R., & Manrodt, K. B. (2003). Forecasting software in practice: Use, satisfaction, and

performance. Interfaces, 90–93.

Schriesheim, C. A., Powers, K. J., Scandura, T. A., Gardiner, C. C., & Lankau, M. J. (1993). Improving

construct measurement in management research: Comments and a quantitative approach for

assessing the theoretical content adequacy of paper-and-pencil survey-type instruments. Journal

of Management, 385–417.

Schwartz, Z., & Cohen, E. (2004). Hotel Revenue-management Forecasting: Evidence of Expert-

judgment Bias. Cornell Hotel and Restaurant Administration Quarterly, 85–98.

62

Singh, I. L., Molloy, R., & Parasuraman, R. (1993). Automation-induced "complacency": Development of

the Complacency-Potential Rating Scale. The International Journal of Aviation Psychology, 111–

122.

Sniezek, J. A., & Van Swol, L. M. (2001). Trust, confidence, and expertise in a judge-advisor system.

Organizational Behavior and Human Decision Processes, 288–307.

Soll, J. B., & Mannes, A. E. (2011). Judgmental aggregation strategies depend on whether the self is

involved. International Journal of Forecasting, 81–102.

Stephens, R. (2013). Essential algorithms: A practical approach to computer algorithms. Indianapolis, IN

: Wiley, [2013].

Szajna, B. (1996). Empirical evaluation of the revised technology acceptance model. Management

Science, 85–92.

Thompson, R. L., Higgins, C. A., & Howell, J. M. (1994). Influence of experience on personal computer

utilization: Testing a conceptual model. Journal of Management Information Systems, 167–187.

Tversky, A., & Kahneman, D. (1975). Judgment under uncertainty: Heuristics and biases. In Utility,

probability, and human decision making (pp. 141–162). Springer.

Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the Qualitative-Quantitative Divide: Guidelines

for Conducting Mixed Methods Research in Information Systems. MIS Quarterly, 21–54.

Venkatesh, V., & Davis, F. D. (2000). A Theoretical Extension of the Technology Acceptance Model:

Four Longitudinal Field Studies. Management Science, 186–204.

Venkatesh, V., Davis, F. D., & Morris, M. G. (2007). Dead Or Alive? The Development, Trajectory And

Future Of Technology Adoption Research. Journal of the Association for Information Systems;

Atlanta, 267–286.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information

technology: Toward a unified view. MIS Quarterly, 425–478.

VERBI Software. (2020). MAXQDA 2020 [computer software]. Berlin, Germany: R Foundation for

Statistical Computing.

63

Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision making: Egocentric discounting and

reputation formation. Organizational Behavior and Human Decision Processes, 260–281.

Yeomans, M., Shah, A. K., Mullainathan, S., & Kleinberg, J. (2017). Making sense of recommendations.

Management Science.

64

Appendix 1: Practitioner Interview Guide

Introduction:

Good morning (afternoon). Thank you for agreeing to meet with me today for an interview. The entire

interview should last approximately 45 minutes to an hour.

The purpose of this study is to understand how decision makers use algorithms in their work. For our

purposes, an algorithm is any well-defined computational procedure that takes some values as inputs and

produces some values as outputs. The procedure identifies both a problem and the steps by which it

should be solved. An example of an algorithm may be a program that takes historical sales data as an

input and makes a prediction on future sales. Another example may be an algorithm that uses your past

search behavior on a website and predicts what you should be shown on the screen.

Recording:

If it is okay with you, I will be recording our conversation. The purpose of the recording is so that I can

get all the details and review your answers in detail later, while at the same time engaging you in the

conversation. All of your comments and responses will remain confidential.

Consent Form:

(If the participant has not already completed the consent form)

Before we get started, please take a few minutes to read this consent form and sign it if you agree to

participate.

(Hand participant the consent form/preamble.)

(After participant returns consent form, turn recording device on.)

Questions

Introduction:

1. To start, I’m interested in learning a bit more about you, the organization you work for, and what you

do in your current role.

a. Tell me about your organization (e.g., products/services, numbers and types of employees

etc.

b. Tell me about your role within the organization (e.g. division/group, reporting

relationships, primary responsibilities, etc.) and what you do in your job.

c. How many years of professional experience do you have?

Key Questions

2. A good way to develop an understanding of how you use algorithms in your job is through

examples. Please tell me about a specific project where an algorithm was used successfully to

improve a decision being made? (If the following are not answered ask:)

a. What was the specific project you were working on?

b. What was the problem the algorithm was designed to address?

c. What was your role on the project?

d. Describe the algorithm? (What was the algorithm? What were the algorithms

inputs and outputs?)

e. Why did you use the algorithm to help make this decision?

65

f. What was the outcome of the use of the algorithm?

3. Thinking about the project you've described, what are the characteristics of the algorithm that

you feel helped you make successful decisions?

4. Please tell me about a time when you were given the opportunity to use an algorithm in a

decision, and you chose not to use that algorithm. (If the following are not answered, ask:)

a. What was the specific project you were working on?

b. What was your role on the project?

c. What was the problem the algorithm was designed to address?

d. Describe the algorithm? (What was the algorithm? What were the algorithms inputs

and outputs?)

e. Why did you choose to use ___ instead of the algorithm?

5. Thinking about the time you’ve described, what are the factors that you take into

consideration to reject the use of an algorithm?

6. When deciding whether or not to use a given algorithm to make decisions, what factors do

you use to judge whether the inputs to the algorithm are appropriate?

7. When deciding whether to use a given algorithms recommendation, what factors do you use

to judge whether the outputs from the algorithm are appropriate?

8. From your perspective, what are the greatest benefits of using algorithms in your job? Why do

you consider these the biggest benefits?

9. In contrast, what do you consider the biggest downsides of using algorithms in your job? Why do

you consider these the biggest downsides?

Probing Questions

That's interesting could you explain that a little more…

Let's see, you said ... just how do you mean that? What do you mean by…

How come…. Could you describe… Can you give me an example of …

Tell me about the … What do you think about…? Was this what you expected?

Could you say some more about that? What do you mean by that . . .?

10. On a scale of 1 to 5 (with 1 being “not an important part of my decision to use an algorithm” to a

5 being “an important part of my decision to use an algorithm”) please rate your willingness to

adopt and use an algorithm to help you make decisions in your job:

a. Past experience with using the specific algorithm

b. Past experiences with using other algorithms

c. Perceived ease of use

d. The nature of the algorithms inputs

e. Perceived usefulness to your job

f. Quality of information obtained from the algorithm

g. Trust in the outputs of the algorithm

h. Subjective norms of the organization

i. The interpretability of the algorithm

j. Knowing who the algorithm was developed by

losing Question

66

11. Is there anything I have not asked that you think is relevant in terms of algorithms and how they

are used in decision making?

END INTERVIEW SCRIPT