black hat masterclass
TRANSCRIPT
Winning your audienceThe DNA of engagement
Identify l Influence l Achieve
APMP UK 14th Annual Conference 2016
Black Hat MasterclassAlex King, Shipley
2
Introduction
INTRODUCTIONBlack Hat Masterclass
3
Introduction
• This presentation will show…
• Preparation is the most important factor in determining the success of a Black Hat
• There are frameworks, techniques and games that can be used to enhance the running of the Black Hat
• The post-event follow-up is essential for turning the Black Hat into actionable intelligence
4
What are Black Hats for?
• The purpose of a Black Hat 1 is to
• Predict competitors’ likely strategies and their strengths/ weaknesses. Suggests strategies to unseat incumbent or defend incumbency
• Validate ghosting issues
• Test competitive effectiveness of capture strategy (some overlap with ‘Blue Team’
1 Larry Newman (2011). Shipley Capture Guide v3.0, p122
5
What are the problems with Black Hats?
• The problems many Black Hat reviews face are
• They don’t make accurate predictions that inform competitive strategy. Many become a talking shop for sharing war stories
• The contributions of the participants are usually very unequal. A minority of participants make a majority of the contribution, not necessarily the minority you want!
• Many of the contributions are not evidence-based. Much of the prediction is based on “me too” / MBA thinking
• They exist to tick a box and the follow-up is non-existent
6
Preparing for the Black Hat
PREPARATION, PREPARATION, PREPARATION
Black Hat Masterclass
7
Five key questions to answer
• As a minimum, the Black Hat should answer the following questions:
1. Who is the strongest competitor, most likely to score the highest combined price and technical score?
2. Which key suppliers are most likely to boost the probability of a competitor winning?
3. What are the lessons from past bids that competitors would have learned?
4. What do we know about the customers buying behaviour• (e.g. historically speaking, to what extend do they buy on Lowest Price vs. Value
for Money)• How would the customer weighting of different assessment criteria favour
different competitors? For example, let’s consider two possible assessment criteria:
• Assessment Criteria A has 30% for Price, 15% for management and 55% for technical. • Assessment Criteria B has 45% for Price, 10% for management and 45% for technical.
• Which competitor is better off in scenario A or scenario B?5. What potential changes to our competitive intelligence would cause
us to nullify our answers to the earlier questions?
8
Black Hat Participants
• The biggest risks to a productive Black Hat will be ‘group think’ and ‘too many cooks in the kitchen’.
• Group think is best avoided by selecting:• We advise 1 junior and 1 senior person for every 3 experienced
participants• 2 technical roles for every 3 or 4 sales and marketing• Do not over-fill the teams, 5-8 people per team. The number of
teams is only limited by your agenda and logistics• Participants should be primarily selected because they have worked
with, for or against the competitors. Rank in the organisation is rarely a good indicator of usefulness to the Black Hat
9
Format of the Event
• A single group discussing the key questions
• Multiple teams role-play as the competitors
• The group discusses key questions and issues, in a facilitated discussion
• The teams are tasked to describe the strategy they would follow for the specific bid
• The key output is a summary of the collective discussion and a bidder comparison matrix
• A separate customer team that evaluates the final presentations
10
Win-Loss Analysis
• A good Win-Loss Analysis identifies all the relevant past bids that can provide evidence of:
• The customers buying behaviour• The tactics and strategy of the competitors• The successes and failures of competing technical solutions
• The purpose of a Win-Loss Analysis document is to be the single source of truth for all these lessons. It should be circulated to attendees 1-2 weeks before the Black Hat in order for them to make comments that improve it.
• We can use the Freedom of Information Act to dig deeper into some of the key bids on our Win-Loss Analysis.
11
Win-Loss Analysis
• Here are some examples of a bid evaluation gained via FOIA, there were 4 bidders, bidders B & C were non-compliant. The contract value is ~£800m and involved the TUPE of ~350 staff.
A
• Most expensive of all bidders• “Insufficient detail over management of cultural change”• “Detail on how relocation would be managed was also lacking”• “Insufficient evidence to demonstrate how [they] would manage retention and re-skilling of existing
staff”• “Little evidence how they would engage with Trade Unions”
D
• “Contain a significant amount of cost and pricing data”• “Provided a very good response on Service Level Management”• “The only bidders to build in surge manpower at the beginning and their understanding of the
requirement was demonstrated by the underlying detail”• “Showed measured consideration of risks and benefits. They backed this up with explanation of their
methodology and tools for achieving this”
12
Bidder Analysis
• After Win-Loss Analysis, we produce a Bidder Analysis, then a Customer Analysis.
• Too much ‘bidder analysis’ is not analysis at all. It is bland statements lifted from promotional material (e.g. annual reports).
• “We have witnessed the benefits of the Group’s long-standing focus on credit quality and the diversification of our income streams which have allowed us to deliver record profits” 2 – RBS, 2007
• The value of Competitive Intelligence is understanding the constraints around bidding, that limit the possible range of behaviours available to competitors.
2 Annual Report and Accounts (2007). Royal Bank of Scotland
13
Base Rate Neglect
• If there are 200 people in the room, then 0.5% are actually infected, 1. This is the base rate.
• If we test 200 people with a 99% accurate test…• Then 1 will test positive who is positive (200 x 0.5% x 99%, round up from 0.99)
• Then 2 will test positive but are negative (200 x (100% - 99%))
• Then 0 will test negative but are positive (200 x 0.5% x (100% - 99%), round down from 0.01)
• So if you have tested positive, there is a 33.3% probability you will turn into a zombie. To remember this, we must remember the base rate, that only 1 person is actually infected.
• A poison has been released into the room. The poison is know to turn 0.5% people exposed to it… into Zombies! Fortunately, we have a 99% accurate test that will determine who is turning. We test everyone in the room. You test positive, what is the probability you are turning into a Zombie?
14
Measuring Culture
3 Culture's Consequences (1980). Geert Hofstede4 The Modern Scholar: Evolutionary Psychology (2010). Allen MacNeill5 The Dynamic Decision Maker (1999). Kenneth Brousseau & Mike Driver
15
Customer Analysis
“Some customers seem unable to justify a 3% percent price premium, while others routinely justify 20%” 6.
6 Winning the Big Ones (2012). Michael O’Guin & Kim Kelly
16
Running the Black Hat
IT’LL BE ALRIGHT ON THE NIGHT
Black Hat Masterclass
17
Porters Four Corners
Porters Four Corners 7 is a superior alternative to the popular SWOT framework. The key difference is that the Four Corners builds up evidence and reasons for strengths and weaknesses, where as SWOT simply presents them in a ‘chicken and egg’ fashion
7 Competitive Strategy (1980). Michael Porter
18
Avoiding ‘Me Too’ thinking
The ‘Decision Styles’ 8 framework is a way to understand how decisions that seem irrational to you, can seem rational to a competitors decision maker.
8 The Dynamic Decision Maker (1999). Kenneth Brousseau & Mike Driver
19
Predictive Markets
Predictive markets work well when they have four characteristics 9:• Diversified Opinions• Independence (between participants)• Decentralisation (people have their own sources of knowledge)• Aggregation
9 The Wisdom of Crowds (2004). James Surowiecki
The actual results of the Conservative Leadership election were: • May 50.2%, Leadsom 20.1%, Gove 14.6%, Crabb 10.3%, Fox 4.9%
20
Post-Black Hat Follow Up
CAN WE TICK THE BOX NOW?
Black Hat Masterclass
21
Confirmation Bias
Iago dislikes Othello for promoting Cassio above him in the venetian army.Iago convinces Othello that Cassio had an affair with Othello’s wife, Desdemona.When Othello confronts his wife, she breaks down crying… which he takes as a sign of her guilt
22
Multiple Hypothesis Testing
Com
petit
or A
Str
ateg
y
Develop new software product
1. In-House Development
Reuse existing software product
2. Buy and Modify
3. Buy Off-The-Shelf
We should define the major options available to competitors, then ask ourselves what evidence would confirm or deny that the strategy was being followed
23
CI Action Plan
We avoid Othello's error by setting out in advance, what evidence we could find that would confirm or deny each strategy.We then set out a ‘CI Action Plan’ to gather evidence and see which strategy it supports
Example of Strategy 3 analysis
24
Keeping Score
Ultimately, our predictions will not get better unless we measure and refine them. We need to know who the ‘foxes’ and ‘hedgehogs’ are 10
10 Superforecasting (2015). Phillip Tetlock
25
Summary
• This presentation has shown…
• Preparation is key. Produce the 3 prep documents, don’t put too many cooks in the kitchen, have a well-defined output template.
• The running of the Black Hat can be enhanced by Porters Four Corners, measurements of Organisational Culture, Decision Styles & Predictive Markets.
• Follow-up the event with Multiple Hypothesis Testing and a CI Action Plan. Most importantly, measure the accuracy of your predictions and continuously seek to improve them.
26
If he knows, that she knows, that he knows
Over 60,000 people were asked to guess a number from 0 to 100 that represents two thirds of the average guess. (whole numbers only, rounded up or down)
If everyone picked a random number, the average guess would be 50, then the player would win by guessing 33. (33 is two thirds of 50)
However, if the average guess then became 33, then the player would win by guessing 22. (22 is two thirds of 33)
27
If he knows, that she knows, that he knows
What the results tell us it that: 4% people didn’t understand the rules and provided impossible answers 22% didn’t really grasp the implications of the rules 21% understood, made one logical step past them then stopped the logic 33% of people were along the right lines 20% assumed too much rationality with the ‘Nash Equilibrium’ assumption (1 or 0)
The correct answer is a weighted average of these groups, 19
