do the algorithms report to the robots?

7/25/2019 Do the algorithms report to the robots?

1/15

DO THE ALGORITHMS REPORT

TO THE ROBOTS?Making decisions in a world of big data; managing in aworld driven by algorithms

Michael RossCo-founder and Chief Scientist, DynamicAction

Computers are useless.

They can only give you answers.PABLO PICASSO


2/15

2 | Big Data Decisions | Do The Algorithms Report to the Robots?

ALGORITHM-AS-EMPLOYEE

Many businesses are getting excited at the possibilities of big data. But data is only useful if it changes the

way you make decisions. Big data may sound exciting, but it is utterly useless if you continue to make

the same decisions, in the same way, with the same data and at the same frequency. Not only will you

have missed the point, you will also have missed the opportunity.

To respond and thrive in this new reality, businesses have to completely rethink how they make decisions, and

then how they execute and manage them.

Algorithms are the answer: the logic of how you make thousands, and even millions, of automated

or semi-automated decisions in the digital commerce world. These algorithms are already amongst us.

Core retail activities such as google paid search, product feeds, product recommendations, on-site search,

retargeting and triggered emails are already powered by algorithms. But just because you have an algorithm,

doesnt mean it is any good.

Algorithms need to be thought of like employees - they need to be managed. They need to be trained,

set the right objectives, reviewed and given feedback. If they consistently underperform, they need to be

replaced.

While humans can be resistant to feedback and may obfuscate or explain away mistakes, algorithms thrive on

feedback and failure. In fact, the essence of machine learning is based on understanding failures and missed

opportunities, and evolving parameters and models accordingly. In the human-computer era, one needs to

establish the right culture, processes and governance that will allow algorithms to develop and thrive.

The management and optimisation of algorithms is a new muscle for retailers. This paper gives practical

guidance on how to rethink business decisions in the digitally-enabled, data-driven, algorithm-powered world.It covers:

1. Why we need to rethink decisions

2. What decisions are we talking about

3. A new approach to making decisions

4. The management challenge


3/15


1. WHY WE NEED TO RETHINK DECISIONS

Decision-making in digital retail, as many retailers have already recognised, is considerably more complex

than in physical retail. Digital commerce is catalysing two tectonic shifts that fundamentally aect decision-

making:

Consumer behaviour is changing radically. Consumers now have almost unlimited choice, and are

empowered and informed. They are using a portfolio of devices, being bombarded by digital marketing

and are shopping in new and complex ways. The result? Consumers create data, providing managers

with a tsunami of available data to use

Businesses have a vast new set of decisions to make. Firstly decisions on how to become digitally

enabled, and thereafter to take advantage of the ability to personalise and optimise all interactions with

customers and products. The result?Decisions consume data, while at the same time creating more

and more data

The consequence is that the logic and economics of many core retail decisions relating to marketing and

merchandising have changed. This is predominantly caused by the transition of marketing from a xed

to a variable cost (per impression, per click, per transaction). Trivial as this sounds, it has changed the

fundamental equations of retail (do I spend the next on price or marketing?). In addition, it has elided the

marketing-merchandising-store organisational boundaries - silos which made sense in the past, however

dont anymore.

At the heart of the change is the atomisation of both the decisions and the data. We are now faced

with millions of decisions and hundreds of millions of data points.

The key is to start with the decision and then ask how the data (whether big, small or medium) allows you to

make decisions more intelligently, more frequently or more protably.

THE CHANGING LANDSCAPE OF DATA DECISIONS

DIGITAL CONSUMER BEHAVIOUR

DECISIONS

DATA

Devices

eCommerce

Beacons

Marketing

Empowered

Informed

Unlimted Choice

Digital enablers

New frequency/logic

Atomised


4/15


2. WHAT DECISIONS ARE WE TALKING ABOUT

Businesses are machines for making decisions: to state the obvious, the day-to-day rhythm of business is

about decisions and actions. The decision architecture for making these decisions has evolved, often over

decades, and is part of the fabric of every retailer. But the digital world is catalysing at an unprecedented pace

and scale of change in this architecture.

Altering decision-making may sound overwhelming as a task, but in reality there are specic types of

decisions that require the most immediate attention and adaptation. High priority should be decisions with a

high value, and where data could have a major impact. A good starting point is to conduct a decision audit:

What decisions are being made at the moment and how are they made? What new decisions will need to be

made in the future?

Decisions can be sorted into a hierarchy, based on the nature of the decision and its frequency. It is then

easier to identify what changes need to be made at each level, and where the greatest dierences will occur.

Change is required across the hierarchy, but the most profound change will be felt at level 3 (tactical

decisions) and level 4 (programmatic decisions), where increased complexity, volume and frequency of

decisions will necessitate some degree of automation.

Level 1. Strategic decisions (annual). Typically familiar decisions that businesses have been making for

years. For example: which new markets to enter, where to locate new shops and what new categories to

develop?

What has changed?- New data is now available to inform the decisions, such as:

- New retail locations can be informed by looking at the postcode of loyal online shoppers

- New categories can be informed by searches made on the site by VIP customers for categories that

do not exist

DECISION HIERARCHY

Small volume of high value decisions

New data needs to be applied

Level 1

Strategic

Level 2

Planning

Level 3

Tactical

Level 4

Programmatic

New approaches are required.

New data and logic required

New frequency of decisions,

combined with new data/logic

Completely new decisions

and constantly evolving


5/15


Level 2. Planning decisions (bi-annual/seasonal). Again, these are familiar decisions such as: range

planning (breadth and depth), which brands and products to continue or delist, marketing budgets;

channel allocation and technology roadmap

What has changed? The decision logic is dierent, requiring new data and logic:

- Size ratio planning can be informed by looking at the size distribution of customers viewing products

and the impact on sell-through- Products can be evaluated in terms of their impact on acquiring and retaining high value customers

- Sales can be planned based on expected sales from existing customers, and consequently how

many new customers need to be acquired

Level 3. Tactical decisions (weekly/daily). Many of these decisions sound familiar, but are actually very

dierent in the digital world. For example: pricing, promotions, digital marketing, site sort orders, landing

pages and digital touch points

What has changed? The decision logic is dierent (new trade-os are required), the level of action is

typically more granular and the frequency of decisions needs to be optimised

- Retailers need to decide (and can now make the trade-o) whether to invest the next in price,

marketing or a customer promotion- CRM actions can be informed by whether the customer is actively browsing the website

- Sort orders can be personalised and changed every week, day or hour. A key question is what is the

optimal frequency?

Level 4. Programmatic decisions (hourly/real-time). For example: programmatic marketing,

personalised search results, keyword bids and product recommendations

What has changed? The decisions are being made at a very low level of resolution, are increasingly in

real-time (so they need to be automated), are often new and continually evolving

- Retailers can suggest unique product recommendations for each customer viewing each product

- Retailers can retarget individual customers who have taken a specic action on their site

- Landing pages can be personalised based on the customer and the intent of their visit

- Site search engines can increasingly personalise the sort order (product ranking) of search results


6/15


3. A NEW APPROACH TO MAKING DECISIONS

Having classied your decisions, how can you make them dierently? Essentially this is about introducing

a more disciplined approach to cope with the volume and complexity of decisions. This is similar to the

business process re-engineering revolution fuelled by computers in the 1980s/90s.

Every decision has a control(what you do) and an objective(what you are trying to optimise):

An algorithm determines how to set the control to optimise the objective

This algorithm requires a model, and implicit in the model is a simulation that will have unknowns

(described by parameters)

Feedback will then improve all aspects of this process from the model and parameters to the algorithm

To help make sense of this, we have developed a decision checklist:

1. Decide what sort of decisionit is (the objectiveand the control) and understand what success

looks like

2. Work out what datais needed

3. Build a modelwith parameters

4. Work out the algorithmto make the decision5. Operationalise - turn the decision into an action

6. Review performance, metricsand feedback

DECISION CHECKLIST

DECISIONS1

ALGORITHM4

ACTION5 OBJECTIVE

FEEDBACK/

METRICS6

DATA2 MODEL +

PARAMETERS3

CONTROL


7/15


Step 1: Decide what sort of decision it is

Many decisions can seem ill-dened, such as managing keywords on google or personalising landing pages.

A critical role for any manager in the digital commerce world is to really understand the decision, and be able

to explain it (often to people without the technical expertise).

For each decision, at every level, it is critical to understand the objective(what are you trying to optimise and

what success looks like). The objective will typically be either:

- Business outcome: prot, revenue, return on capital or customer LTV

- Interim outcome: product views, new customers acquired or achieved margin

It is also critical to understand the controls(what decisions are actually possible). At a high level, decisions

can be categorised into four main types: binary, discrete, threshold or value (see table for further detail).

Being clear on the controls can help make sense of the decision, and then more clearly articulate it to others.

Moreover, each type of control requires a dierent approach to management and feedback.

Control Description Examples

Binary choice Decisions that have a yes/

no answer

Do we open or close a store?

Do we list or delist a brand?

Do we accept or reject a potentially fraudulent order?

Discrete choices Decisions where there are

a specic set of answers to

choose from

Do we mark down a product by 30/50/70%?

How often should we change our website

homepage?

How many dierent emails should we send per week?

Threshold The decision is a number

where we will take a

dierent action above/

below the value

Whats our maximum acceptable threshold for cost

per order on google?

Whats the annual spend threshold for a VIPcustomer?

Whats the value threshold for free delivery?

Value The decision is a specic

value

How many units do we order of a product?

How much should we bid on a paid search keyword?

How much should we spend to acquire a new

customer?

As well as understanding the objective and controls of each decision, it is important to work out the

interaction between them:

What frequency of control is possible? If wrong, how quickly can the decision be changed? Whats the feedback time? How quickly will we know if the decision is good or bad?

Whats the cost of a wrong decision?

Whats the sensitivity of the outcome to a bad decision?

This is critical to set the context for making the decision. It is easy to change your mind on decisions that are

low cost and rapid feedback, but more expensive to reverse decisions which are high cost and where feedback

is slow.


8/15


Step 2: Work out what data is available and needed

There is an almost unlimited amount of data available to make these decisions. The challenge is to work out

what data you really need, and how this might evolve as the decision gets more valuable and the data gets

cheaper.

Figuring this out requires an implicit data-decision trade-o: is it better to make an OK decision with easy

data or a perfect decision with complex data? In practice, the key is to make the trade-o and make a

good enough decision. (This is the essence of satiscing an approach to decision making called bounded

rationality, which recognises that humans are limited in their ability to determine the optimal solution to a

decision. Herb Simon developed the concept in 1956 and won a Nobel Prize for it in 1978).

To make decisions eectively, businesses need to think about making good enough decisions based on the

minimum amount of data required. It is critical to avoid asking for all the possible data available that might be

useful. Or you will quickly become overwhelmed and suer from crippling analysis paralysis.

For managers to eectively make this data-decision trade-o it is important to really understand:

What data is readily available vs. hard to get?

What data is available now, and how might it evolve?

How good is the data? (e.g., cleanliness, completeness, ability to join, quality)

What data is free vs. at a cost?

How far can you get with siloed data? Whats the value of joining data?

How might the decision logic evolve as new data becomes available?

What the ROI of investing in better data?

EXAMPLE: DATA-DECISION TRADE-OFF FOR PAID SEARCH MANAGEMENT


9/15


Step 3: Build a model with parameters

At the heart of the new approach to decisions is a model of how the world works (one can also think of the

model as describing the rules of the game). For each decision it is critical to understand what you expect to

happen if you make the decision.

In alldecisions, there is some type of uncertainty, and the model and parameters describe this uncertainty.

The uncertainty is typically characterised by one or more parameters, and embedded in the models will

typically be an unknown what-if relationship - if we do X, we hope/expect that Y will happen. For example:

- How will volume increase if we reduce price? (price elasticity)

- How will visits/sales increase if we increase marketing? (marketing elasticity)

- How will customers behave if we ship orders faster? (customer experience elasticity)

In all decisions, there is also some type of trade-o, a cost and a benet. The model captures the natureof

the trade-o, and the parameters describe the scale of the trade-o. For every decision, there will be one

model that then gets applied by the algorithm (see below) across the thousands/millions of programmatic

decisions.

The key to success is to recognise that a critical aspect to the management role is setting, monitoring

and managing parameters and then evolving the model.

As many managers will recognise, it is incredibly easy to make decisions without having a model or simulation.

In practice, one can get away with this if the business performs well (you could get lucky!). But it is a risky

proposition in the long term because when things dont go so well, it is then hard to know what went wrong,

and what you should do dierently.

A SIMPLE MODEL: WHAT IF?


10/15


Step 4: Work out the algorithm

The algorithm describes how the decision is actually made. Its inputs are the data, the simulation and the

objective. The algorithm answers the question what value of the control will optimise the objective?

The algorithm here is typically described by an equation (or logic statement) that connects the decisionto the objective. Sometimes, one can simply write down the algorithm as a formula (e.g., a replenishment

algorithm); alternatively, some algorithms will be executed using some sort of optimisation routine (e.g.,

nding an optimal sort order).

ALGORITHM EXAMPLE: BIDDING ALGORITHM FOR KEYWORDS

The essence of algorithms is that they operate at a low level of resolution, so one algorithm might make

thousands, if not millions, of programmatic decisions. Algorithms are everywhere. Whether retailers like it

or not, more and more of the critical decisions within their digital commerce business are increasingly being

made by algorithms.

THE STATE OF THE ALGORITHM - EXAMPLES

Example decisions Control Level of execution Algorithm example

How products areranked when an on-site

search is made

Value Score for every product, forevery search (millions)

Retailer formula based oninventory, sales, conversion,

newness, availability, etc.

How orders are

accepted or rejected as

potential fraud

Threshold All orders are scored

based on a combination of

customer and order data,

and 3rd party fraud systems

Accept order if score >X, reject

order is score


11/15


Step 5: Operationalise - turn the decision into an action

It is an easy assumption that actions somehow happen automatically in the digital world. In fact, many things

can go wrong when decisions become actions. Since the action is typically not observable, it is often hard to

distinguish between:

- Execution failure the action simply has not happened- Poor execution the action has happened, but has been badly executed (e.g., human error)

- Unexpected outcomes the action has been properly executed, but results in some sort of technical

failure

Turning a decision into an action should be straightforward, but often it isnt. So it is important to really

understand:

- Who takes the action? And what happens if they are on holiday?

- How often do they take it?

- How complex is the task or action?

- How is the execution managed (if not observable)?

In traditional human decision-making, the RAPID framework (a variant of the RACI framework) is a helpful way

to understand who does what in making decisions and taking action. While still helpful in the new human-

computer world, it is critical to rethink this approach where some combination of the R, A, P, I and D will be

driven by an algorithm. For the humans involved in this process, its critical to ensure they have the right

information, support and tools required.

WHAT DOES RAPID LOOK LIKE IN AN ALGORITHM-DRIVEN WORLD?

Need a clear owner forthe algorithm

Need someone tomanage the execution

Need a governanceprocess to review the

algorithm


12/15


Step 6: Monitor and Review Performance

How do you manage the algorithm-as-employee? A key challenge for any manager (and perhaps also the

Chief Algorithm Ocer) is to work out what is going wrong and why it is happening. Reviewing performance

and managing the algorithm requires a portfolio of metrics; metrics for what you did and what happened vs.

what was expected in terms of prot, wastage, long term consequences or unintended consequences.

At a high level, one needs to have an overall measure of performance vs. the objective and then understand:

How did the decision perform overall? How has the decision performed vs. what was predicted? If

performance was good, we can focus on improvement. If performance was bad, we need to understand

whats going wrong:

- If the prediction is good but performance is bad, it is likely theres a problem with the objective i.e.

weve optimised to an objective that is not correlated with prot

- If performance is good but the prediction was bad, we need to review the model and parameters

Managing the model, parameters and algorithm requires a disciplined approach to understanding decision

quality. For every decision, one has to dene an error which could be: wasted spend, wasted eort or

missed opportunity (a measure of whats gone wrong, or what could be better). One can then look at the

distribution of errors to understand where to focus. For example, if the decisions have a:

- Consistent errorgreview parameters. Likely that tweaking a parameter will solve the

problem

- Inconsistent errorgreview model. Likely that the model doesnt represent reality

- Concentrated errorgreview algorithm. Likely that performance can be improved by making

the algorithm more specic (e.g., if errors are concentrated with specic customers/categories/

outliers)

Much like employees, algorithms have a learning curve. But unlike employees, algorithms thrive on learningfrom mistakes (its the essence of all machine learning). When it comes to managing algorithms, leaders

have to celebrate failure (or at least encourage openness). Its too easy to present aggregated averages that

obfuscate the issues and opportunities.

MANAGING THE ALGORITHM: GOOD THINGS TO MEASURE


13/15


4. The management challenge: co-ordinating all the decisions

We are in the midst of a digital industrial revolution and the before leaders navigating this sort of digitally-

led, data-driven transformation face a daunting task. Managing a business with hundreds (or thousands) of

algorithms is hugely complicated. We recommend three critical foundations for success: data, people and

culture.

DATA

Data is often described as the new oil, needing to be mined, extracted and rened. Unfortunately, it is

often an afterthought because the consequences of bad data are not immediately visible. But bad data is

cancerous quietly undermining the quality of decisions.

In reality many retailers are suering from signicant data debt, where years of poor (or non-existent)

data management are now coming home to roost. To introduce good data into your business, we suggest

focusing on the three core building blocks:

Collecting data: Businesses need to recognise the criticality of instrumentation to ensure that data

is collected and is of high quality. A common excuse from retailers is our analytics tagging is not very

good, but when web analytics is the critical glue that joins data in the digital world this excuse should no

longer be tolerated

Joining data: While the data all originates in silos, it has to be joined to have any hope of making prot-

centric decisions. The key to joining data is ensuring common elds between systems

Storing data: Data storage has historically been seen as a cost to be optimised, and so data has

been aggregated or thrown away. But storing data is now almost free - Amazon Web Services has

commoditised storage (for example standard storage is now 3 cents per month for 1 GB)

THE INSTRUMENATION CYCLE

ACTIONS DECISIONS

CUSTOMER

BEHAVIOURMANAGEMENT

DATA

Humans

Computers

Workflow

Tagging

Alogorithms

Logic

Digital exhaust

Transactions

Metrics

Inference

Organisation

Incentives

Automation cycle

Intrumentation cycle Optimisation cycle


14/15


PEOPLE

It is ironic that in a world of big data and automation, many of the toughest management challenges are

people-related. While many aspects of retail decisions can be automated, people are still needed at the

heart of the business it is people who turn the data into something useful.

Retailers need to employ many dierent types of people to make sense of their data, and it is critical to think

of data as a team eort. Some of the roles we are now seeing in successful data-centric businesses include:

Decision architects: people who understand the business problem you are trying to solve, and can work

out what questions to ask. This is the creative end of data science and these are typically people with a

strategy consulting background

Algorithm developers: people who understand the logic/math to translate the business problem into a

solvable math problem. The key here is to rene the question into something that is answerable given the

available data. These people are typically mathematicians, statisticians and engineers

Data analysts: people who are able to make it work once. These are people with advanced Excel, basic

database and web analytics skills, who are good at cleaning, joining and manipulating data

Data product managers: people who understand the technical requirements to translate the one-o

solution into a technical roadmap. These are people with technical product management experience and

they are the bridge between the business and developers

Big data developers: people who can actually build and operationalise a technical solution. These are

people with skills in database architecture and software development

WHAT SORT OF DATA PROBLEM ARE WE TRYING TO SOLVE

Decisionscience/project

management

Automation/artificial

intelligence

Exploratorydata mining

Reporting/

businessintelligence

Action-focus

Businessproblem

Business process

Insight-focus

One-off Operational


15/15


CULTURE, PROCESS AND GOVERNANCE

Digital commerce does not respect organisational boundaries. The inconvenient truth is that many of the

critical decisions of digital commerce are confounded by data from outside the system or organisational silo.

The data creates the challenge and people are a critical part of the solution - but only if the right structure,

process and governance is put in place.

As a result, it is inevitable that some of the traditional ways of managing will need to change. What change

is needed clearly varies across organisations, but will typically require retailers to rethink some or all of the

following:

- Behaviours: Are we encouraging people to ask the right questions?

- Skills: Do we have the capability to answer?

- Resource: Do we have the means to action?

- Incentives:Are people correctly incentivised?

- Control: Can we measure success?

- Organisation: Are the right people making decisions at the right level?

The transition to a world of algorithm-driven retail is a huge challenge for all retailers, but the opportunities areequally huge for the winners. At the core of this transition is the cultural shift to make an organisation think

dierently. Too often retailers revert to making the same decisions, on the same frequency with the same

logic as theyve done in the past. This is not a formula for long-term success.

The role of humans at Amazon is to help computers make better decisions.Attributed to Je Bezos, CEO of Amazon.com

do the algorithms report to the robots?

Documents