aide memoire on - dln-rad.forces.gc.ca
TRANSCRIPT
Page 1 of 113
Aide Memoire on
Intelligence
Analysis Tradecraft
UNCLASSIFIED
For training
purposes only.
Version française disponible.
Canadian Forces
Intelligence Command
Version 6.00
August 2015
Page 2 of 113
Aide Memoire on
Intelligence Analysis Tradecraft
Version 6.0 August 2015
Table of Contents
INTRODUCTION ....................................................................................................................................................... 6
BEST PRACTICES FOR INTELLIGENCE ANALYSTS .................................................................................................. 6 ELEMENTS OF ANALYSIS ......................................................................................................................................... 7
Evidence (Uncertainty). ....................................................................................................................................... 7 Assumptions. ........................................................................................................................................................ 7 Factors. ................................................................................................................................................................ 8 Assessing the Future. ........................................................................................................................................... 8 Drivers and Criteria. ............................................................................................................................................ 9 Assessments, Scenarios and Hypotheses. ............................................................................................................. 9
THE INTELLIGENCE CYCLE ................................................................................................................................... 10 LEVELS OF INTELLIGENCE SUPPORT .................................................................................................................... 10 CATEGORIES OF INTELLIGENCE ............................................................................................................................ 12 TYPES OF STRATEGIC ANALYSIS PRODUCTS ........................................................................................................ 12 ANALYTIC RIGOUR ................................................................................................................................................ 13 TEMPORARY POSTMODERNISM ............................................................................................................................. 13
REFLECT ON THE PROBLEM, DETERMINING POSSIBLE APPROACHES. ............................................ 14
WHAT IS THE QUESTION? ...................................................................................................................................... 14 W5H & SW. ........................................................................................................................................................ 14 (W5H & SW)C4 .................................................................................................................................................. 15 Policy Prescription. ........................................................................................................................................... 15
WHAT IS THE NATURE OF THE QUESTION? ............................................................................................................ 15 Puzzles, Mysteries and Messes. .......................................................................................................................... 15 Three Kinds of Questions. .................................................................................................................................. 16 How much detail? .............................................................................................................................................. 16 Time. .................................................................................................................................................................. 16 Client Knowledge. .............................................................................................................................................. 16
ISSUE REDEFINITION .............................................................................................................................................. 17 IDENTIFYING FACTORS .......................................................................................................................................... 17 SUMMARY ............................................................................................................................................................... 17
BE RESOURCEFUL AND SYSTEMATIC WHEN COLLECTING INFORMATION, DOCUMENTING
SOURCES AND NOTING CAVEATS ON USAGE. ............................................................................................. 18
GUIDING FACTORS ................................................................................................................................................. 18 INT COMPARISON CHART: PROS AND CONS ........................................................................................................ 18
OSINT................................................................................................................................................................ 19 HUMINT ........................................................................................................................................................... 19 SIGINT .............................................................................................................................................................. 20 IMINT................................................................................................................................................................ 20
COLLECTION PLANNING ........................................................................................................................................ 21 Requirements. ..................................................................................................................................................... 21 Indicators. .......................................................................................................................................................... 22 Collection Plans. ................................................................................................................................................ 23
DATA SORTING ....................................................................................................................................................... 26
Page 3 of 113
DOCUMENTATION OF SOURCES ............................................................................................................................. 26 Annotated Bibliography. .................................................................................................................................... 26
ROTBERG, ROBERT I. WHEN STATES FAIL – CAUSES AND CONSEQUENCES. PRINCETON: PRINCETON UNIVERSITY
PRESS, 2004. ISBN 0691116717 (DG’S LIBRARY) (UNCLAS) ................................................................................. 27 Keep track on-line. ............................................................................................................................................. 27 Analysts Notebook. ............................................................................................................................................. 27 Cornell Note Taking System. .............................................................................................................................. 28
CLASSIFICATION MARKING ................................................................................................................................... 28 Classifications. ................................................................................................................................................... 30 Control System Marking. ................................................................................................................................... 31 Sub-Control Systems. ......................................................................................................................................... 31 Dissemination Control Marking......................................................................................................................... 31
AUDIT TRAIL .......................................................................................................................................................... 31 SUMMARY ............................................................................................................................................................... 31
CRITICALLY EVALUATE THE QUALITY OF ALL INFORMATION ......................................................... 32
DATA TRIAGE ......................................................................................................................................................... 32 DATA DIAGNOSTICS ............................................................................................................................................... 32
Provider Diagnostic. .......................................................................................................................................... 32 Information Diagnostic. ..................................................................................................................................... 34 Relevance Check. ............................................................................................................................................... 35 Denial and Deception. ....................................................................................................................................... 37
DATA QUALITY DIAGNOSTIC ................................................................................................................................. 38 PROPAGANDA ......................................................................................................................................................... 39
How to identify propaganda - Red Flags ........................................................................................................... 39 CHECKLISTS FOR EVALUATING INFORMATION .................................................................................................... 40
HUMINT Reporting ........................................................................................................................................... 40 Alpha-Numeric Source-Data Rating .................................................................................................................. 40 Four Step - Web Page Evaluation Checklist. ..................................................................................................... 40
DEVELOP MULTIPLE HYPOTHESES / EXPLANATIONS ............................................................................. 42
FORMULATING HYPOTHESIS ................................................................................................................................. 42 Black Swan Rule................................................................................................................................................. 42 Value Added. ...................................................................................................................................................... 42 Three Basic Approaches. ................................................................................................................................... 42 Types of Hypotheses. .......................................................................................................................................... 43 A Good Hypothesis ............................................................................................................................................. 43
HYPOTHESES (AND SCENARIO) GENERATION ....................................................................................................... 43
CHALLENGE ASSUMPTIONS, MINDSETS AND BIASES............................................................................... 44
ASSUMPTIONS ......................................................................................................................................................... 44 Implication of Assumptions. ............................................................................................................................... 44 Key Assumptions Check. .................................................................................................................................... 44
MINDSETS AND BIASES ........................................................................................................................................... 45 Implication of Mindsets. ..................................................................................................................................... 45 Implication of Biases. ......................................................................................................................................... 46
COUNTERING MINDSETS AND BIASES ................................................................................................................... 46 PREVENTION OF FIXED MINDSETS ....................................................................................................................... 47
BUILD COLLABORATIVE NETWORKS ............................................................................................................ 48
CANADIAN SECURITY AND INTELLIGENCE COMMUNITY ..................................................................................... 48 WHY COLLABORATE? ........................................................................................................................................... 48 CREATING A COLLABORATIVE CULTURE ............................................................................................................. 49 TEAM/GROUP COLLABORATION ........................................................................................................................... 50
Models of Collaboration. ................................................................................................................................... 50 Developing Effective Teams. .............................................................................................................................. 51
Page 4 of 113
Team Formation. ................................................................................................................................................ 51 CONSENSUS VS DISSENT ......................................................................................................................................... 52
Common Pitfalls with Small Groups. ................................................................................................................. 52 Benefitting from Diversity. ................................................................................................................................. 53 Advocacy vs Objective Inquiry. .......................................................................................................................... 53 Forcing a Consensus. ......................................................................................................................................... 54 Encouraging Consensus. .................................................................................................................................... 55
USE STRUCTURED ANALYTIC TECHNIQUES ............................................................................................... 56
WHY USE STRUCTURED TECHNIQUES? ................................................................................................................. 56 CATEGORIES OF TECHNIQUES ............................................................................................................................... 56 STRUCTURED ANALYTIC TECHNIQUES MAP ........................................................................................................ 57 USE WORK GROUPS ................................................................................................................................................ 58
Team A / Team B. ............................................................................................................................................... 58 Red Team. .......................................................................................................................................................... 59 Red Cell. ............................................................................................................................................................. 60 Devil’s Advocacy. ............................................................................................................................................... 61
DEVELOP ................................................................................................................................................................. 62 Structured Brainstorming................................................................................................................................... 62 Delphi Method.................................................................................................................................................... 63 Environmental Scanning. ................................................................................................................................... 64 Outside-In Thinking. .......................................................................................................................................... 65 Hypothesis Generator. ....................................................................................................................................... 66 Hypothesis Review Technique. ........................................................................................................................... 67 Reframing the Question...................................................................................................................................... 68
EVALUATE ............................................................................................................................................................... 69 Data Diagnostic. ................................................................................................................................................ 69 Key Assumptions Check. .................................................................................................................................... 70 Indicators of Change. ......................................................................................................................................... 72 Indicators Validator. .......................................................................................................................................... 73 Analysis of Competing Hypotheses. (ACH). ...................................................................................................... 74
EXPLORE NETWORKS .............................................................................................................................................. 75 Social Network Analysis. .................................................................................................................................... 75 Link Charts. ........................................................................................................................................................ 76
COMPARE ................................................................................................................................................................ 77 Cross-Impact Matrix. ......................................................................................................................................... 77 Structured Comparison. ..................................................................................................................................... 78 Weighted Rankings. ............................................................................................................................................ 79 Change Analysis. ................................................................................................................................................ 80
GENERATE SCENARIOS ............................................................................................................................................ 81 Cone of Plausibility. ........................................................................................................................................... 81 Quadrant Hypothesis Generation Technique. .................................................................................................... 82 Alternative Futures Analysis. ............................................................................................................................. 83
UNDERSTAND POSSIBILITIES ................................................................................................................................... 84 Force Field Analysis. ......................................................................................................................................... 84 Counterfactual Reasoning. ................................................................................................................................. 85 Bow-Tie Technique. .......................................................................................................................................... 86
DEMONSTRATE ........................................................................................................................................................ 88 What If? Analysis. .............................................................................................................................................. 88 Chronologies and Timelines. ............................................................................................................................. 89 Mind Mapping. ................................................................................................................................................... 90 Matrices. ............................................................................................................................................................ 91 Decision Trees.................................................................................................................................................... 92 High Impact - Low Probability. ......................................................................................................................... 93
REVIEW PRE-PUBLICATION ..................................................................................................................................... 94 Dialectic Inquiry. ............................................................................................................................................... 94
Page 5 of 113
Pre-Mortem Assessment. .................................................................................................................................... 95 MATCHING THE TECHNIQUES TO THE TASKS ....................................................................................................... 96
WRITE CLEAR, CONCISE, AND CLIENT FOCUSED REPORTS .................................................................. 99
CFINTCOM ANALYTIC PRODUCT STANDARDS .................................................................................................. 99 PRESENTATION OF ANALYSIS .............................................................................................................................. 100 LANGUAGE OF UNCERTAINTY ............................................................................................................................. 101 EXPRESSING ANALYTIC CERTAINTY ...................................................................................................................... 102
Provide strong judgements ............................................................................................................................... 102 Likelihood ........................................................................................................................................................ 102 Analytic confidence .......................................................................................................................................... 103
FOOTNOTES .......................................................................................................................................................... 104 THE REASONING PROCESS .................................................................................................................................. 105
Deductive Reasoning. ....................................................................................................................................... 105 Inductive Reasoning. ........................................................................................................................................ 105 Abductive Reasoning. ....................................................................................................................................... 105
FALLACIES............................................................................................................................................................ 106 PRE-ATIP PROCESSING ....................................................................................................................................... 106
THE TWO-HOUR CHALLENGE ........................................................................................................................ 108
BEST PRACTICES FOR INTELLIGENCE ANALYSIS REVIEW ................................................................. 110
ANALYTIC RIGOUR MAP .................................................................................................................................. 113
The Aide Memoire will be a work in progress for some time.
Anyone with suggestions as to how to improve this version should send their
comments to Gudmund Thompson ([email protected]) or Ramine
Shaw ([email protected]), noting the version number to which the
comments refer.
Page 6 of 113
Introduction
The purpose of this Aide Memoire is to help defence intelligence analysts and managers to
produce clear, concise, insightful, and client focused analytic reports. While the material herein
may be useful for investigators (criminal, counter-intelligence, etc.), technical and/or scientific
analysts, the Aide Memoire is specifically intended to help guide Assistant Chief of Defence
Intelligence analysts working in the Regional and Transnational Directorate. As a result, the
material will not focus on the forensic, technical, scientific, or investigative aspects of analysis,
but rather on the tradecraft that will help unravel the difficult and complicated issues of
understanding and interpreting human motivations, intentions and future actions.
Analysts being guided by this Aide Memoire will need to be able to move beyond deduction and
induction, as the dominant descriptor of their reasoning process, to abduction or inference to the
best explanation.
Best Practices for Intelligence Analysts1
The Aide Memoire is structured to follow the Best Practices for Intelligence Analysts, developed
by the Intelligence Analyst Learning Program.
The Best Practices are reviewed in some detail starting on page 110.
1 Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analyst Learning Program.
Best Practices for Intelligence Analysts
1. Reflect on the problem, determining possible
approaches. 2. Be resourceful and systematic when collecting
information, documenting sources and noting caveats on usage.
3. Critically evaluate the quality of all information. 4. Develop multiple hypotheses / explanations. 5. Challenge assumptions, mindsets and biases. 6. Build collaborative networks. 7. Use structured analytic techniques. 8. Write clear, concise, well-documented, and client-
focused reports.
Page 7 of 113
Definitions
Elements of Analysis
Analysts wishing to study how to produce and communicate insightful and forward looking
analysis will be confronted by a seemingly unending, duplicative and contradictory list of terms.
In order to deal with and differentiate between these various elements of analysis, this Aide
Memoire will use the terms evidence (uncertainty), assumptions, factors, criteria, drivers,
assessments, scenarios and hypotheses. These terms are defined below.
Evidence (Uncertainty).
Evidence is any available circumstance, piece of data, information or research, or analytic
reflection, etc. that supports, influences or contributes to a belief, proposition, or analytic
assessment.
Sometimes evidence can be considered as factual, or
irreproachable, especially in the scientific, technical and
forensic realms. Generally, however, and especially in
analysis that deals with the human condition, capabilities
or intentions, evidence should be dealt with as
embodying an element of uncertainty. From an analytic
perspective, evidence should almost always be
understood as occupying a position along an uncertainty
spectrum or range, where one end describes the lowest
possible value (zero percent in the graphic on the right) and
the other end the highest possible value (100 percent in this
example). By dealing with evidence as an uncertainty that
occupies an identified position along a range, analysts will
(or should) be open to the possibility that the selected
position on the spectrum may not be exactly correct.
Evidence is generally more compelling when dealing with
things that have already happened (a telephone intercept may
indicate that an individual was implicated in a plot), than it is when trying to discern the future
(an equivalent telephone intercept is likely less compelling when contemplating an individual’s
potential role in an potential, possible or upcoming plot).
Assumptions.
Sometimes, in analysis, the uncertain characteristic of evidence, or indeed the
lack of conclusive evidence, produces a debilitating stalemate that causes the
analytic process to stall. In these cases, it is often helpful to decide to “take
something for granted” in order to allow the analysis to proceed.
An assumption, then, is defined as something that is taken for granted, or
or
Assumption
or
Assumption
Evidence (Uncertainty)
100%0%
Evidence (Uncertainty)
100%0%
Evidence (Uncertainty)
When considering the fuel
efficiency of a new car, an
analyst would understand that
the indicated 4.2 litres/100 km
may actually be 4.6 or indeed
possibly 3.8 litres/100 km.
Page 8 of 113
accepted as being true, without proof, for the purpose of argument or action. In an intelligence
context, an assumption allows some notion or idea to be
considered as fact, for the purpose of the analysis. A
fundamental characteristic of an assumption is that it either
is, or it isn’t!
In good intelligence analysis, assumptions, especially
assumptions that are considered key to the analytic line, need
to be specifically identified, in order to warn the reader that
the conclusions are contingent on the veracity of the
assumption.
Factors.
Factors play a very important role in analytic assessments, especially predictive ones, as they
allow the analyst to describe or contemplate influential elements without being constrained by
the specificity of evidence, or assumptions. Factors,
like evidence, can be measured, that is they can be
understood as incorporating a range (low to high), but
unlike evidence, where the analyst is expected to
make a determination as to where on the range the
evidence rests, or an assumption, which either is or
isn’t, a factor’s real value lies in its ability to facilitate
comparison when taken as a whole. This comparison
is made possible when the analyst identifies the level
of influence (weak to strong) that a factor has in the
particular analysis being contemplated. Exactly the
same factors may have markedly different levels of
influence in different situations.
Assessing the Future.
In producing predictive analysis dealing with future
issues, various approaches are available. The analyst
may identify specific pieces of evidence that are
expected to be important, and use an assessment of their likely position on the uncertainty scale
to predict the future. Or, the analyst can simply identify a number of assumptions and predict the
future based on them. Or, the analyst might identify the factors that will be influential in
determining how the future will unfold; and then, by examining the likely influence of each of
the factors on the issue at hand, produce an assessment of what the future will likely hold. This
latter approach is considered to be the best in many circumstances.
This is not to say, in situations where something has happened regularly for a considerable
period of time, that an assumption that this will continue cannot be useful in assessing the future.
Further, where evidence is strong and well understood, it is reasonable to use that understanding
to predict the future. However, in both cases, the good analyst will look for other factors that
Assumption
An analyst contemplating the
purchase of a new family car
may want to identify as a key
assumption the expectation
that the family unit (husband,
wife, two kids) will remain
intact throughout the process.
Factor
Gas consumption and reliability may
both be factors in the purchase of a
new car. Gas consumption is
measurable (1 to 25 litres per100 km)
as is reliability (1- to 5-star rating),
but their influence in the decision
making process, when they are
considered as a whole, may be very
different. A pensioner may feel that
gas consumption is far more
important than would someone that
has just secured a high paying job.
Similarly, a new mother may think
that reliability is critical to her car
purchasing decision.
Page 9 of 113
Evidence (Uncertainties) Factors
Assumption
Hyp
oth
esis
Hyp
oth
esis
Hyp
oth
esis
Assumption
Hyp
oth
esis
Scenario (s)
Assessment
Evidence (Uncertainties) Factors
Assumption
Hyp
oth
esis
Hyp
oth
esis
Hyp
oth
esis
Assumption
Hyp
oth
esis
Scenario (s)
Assessment
may become influential in determining whether or not the future will look like the past and base
the analysis on this broader understanding.
Drivers and Criteria.2
A driver is a factor whose influence is significantly
more dominant than others. There can be several
drivers in a complex situation, and the term driving
factor may be used. Clearly identifying drivers or
driving factors in analysis is often very useful.
A criterion is also a factor, but it generally embodies
a specific standard against which something can be
judged or decided. A criterion is a condition that
either must or must not be fulfilled for the assessed
scenario to exist, making it (like an assumption)
binary.
Assessments, Scenarios and Hypotheses.3
An assessment is an analytic position. It may arise directly from evidence or a factor or two
(indeed, it may arise directly from an assumption!). Generally, however, assessments are
informed by scenarios or hypotheses (even if they are unstated) – which in turn rely on evidence,
factors and assumptions.
A scenario is an imagined or projected sequence
of events that can be used to demonstrate how a
potential future or specific end-state could come
about. In order to be effective, scenarios must
be based on a “realistic” interpretation of events
– not a fanciful or imaginative one that the
intelligence client may interpret as
unreasonable.
A hypothesis is a proposition set forth as an
explanation for the occurrence of some
specified event or phenomena. A hypothesis
may explain a scenario, or several hypotheses
may describe separate aspects of a single
scenario. Hypotheses can be disproven (the
Black Swan phenomenon – see page 42), but they can rarely be proven. If a hypothesis is
disproven, it does not necessarily mean that the scenario it supports in invalid, though the
scenario would likely need to be adjusted to compensate.
2 Based on work done by Maj Richard Little, 2014.
3 Based on work done by Maj Richard Little, 2014.
Factors, Drivers and Criteria
In an analysis regarding the purchase
of a new car, factors may include:
- gas consumption,
- cargo capacity,
- safety, and
- reliability.
A driver, or driving factor, would
almost certainly be price, while, for a
family with small children or an
aging parent, 4-door might be a
criterion.
Page 10 of 113
The Intelligence Cycle
While various intelligence agencies
and authors have made numerous
modifications and/or additions to
the intelligence cycle, it remains
the foundation upon which all
intelligence work is done. The
intelligence cycle is generally
thought to consist of four phases:
1. Identify your intelligence
needs.
2. Collect the information.
3. Analyse the information
4. Report the information.
There are, however, numerous other interpretations. The one used to describe intelligence-led
policing is shown below.
PRIORITY
SETTING
DIRECTION &
PLANNINGCOLLECTION
EVALUATION
ANALYSIS DISSEMINATIONDIRECTION &
PLANNINGINVESTIGATION
ARREST
DISCLOSURE &
PROSECUTION
IMPACT
ASSESSMENT
INTELLIGENCE
CYCLE
ENFORCEMENT
CYCLE
COLLATION
PRIORITY
SETTING
DIRECTION &
PLANNINGCOLLECTION
EVALUATION
ANALYSIS DISSEMINATIONDIRECTION &
PLANNINGINVESTIGATION
ARREST
DISCLOSURE &
PROSECUTION
IMPACT
ASSESSMENT
INTELLIGENCE
CYCLE
ENFORCEMENT
CYCLE
COLLATION
PRIORITY
SETTING
DIRECTION &
PLANNINGCOLLECTION
EVALUATION
ANALYSIS DISSEMINATIONDIRECTION &
PLANNINGINVESTIGATION
ARREST
DISCLOSURE &
PROSECUTION
IMPACT
ASSESSMENT
INTELLIGENCE
CYCLE
ENFORCEMENT
CYCLE
COLLATION
PRIORITY
SETTING
DIRECTION &
PLANNINGCOLLECTION
EVALUATION
ANALYSIS DISSEMINATIONDIRECTION &
PLANNINGINVESTIGATION
ARREST
DISCLOSURE &
PROSECUTION
IMPACT
ASSESSMENT
INTELLIGENCE
CYCLE
ENFORCEMENT
CYCLE
COLLATION
1. Identify
your intelligence
needs.
2. Collect
the information.
3. Analyse
the information.
4. Report
the information.
1. Identify
your intelligence
needs.
2. Collect
the information.
3. Analyse
the information.
4. Report
the information.
1. Identify
your intelligence
needs.
2. Collect
the information.
3. Analyse
the information.
4. Report
the information.
1. Identify
your intelligence
needs.
2. Collect
the information.
3. Analyse
the information.
4. Report
the information.
Page 11 of 113
Levels of Intelligence Support
It is generally accepted that intelligence analysis
can be separated into three levels, though the
specific definitions of these levels is often the
source of heated debate. For the purposes of
this aide memoire, the three levels of
intelligence are defined as follows:
Strategic Intelligence – is required for the
formulation of policy and plans at the
departmental/agency, national, or
international level. This is the highest level
intelligence, and as such should be all-source, estimative, and focused on future or
comprehensive threats.
Operational Intelligence – is required for the planning and preparation for future operations
and/or investigations.
Tactical Intelligence – is required for the planning and execution of ongoing operations.
Nothing says, however, that these levels need to be symmetrical. Indeed, by separating each
level of intelligence into an operating environment and intelligence support, one can envisage a
very asymmetric juxtaposition, where each level of intelligence support provides input to all
levels of the operational environment.4
4 Developed from “a conceptual model of military intelligence in the contemporary operating environment”
proposed by Jim Cox in his doctoral thesis, Apr 2009.
Strategic
Operational
Tactical
Strategic
Operational
Tactical
Strategic
Operational
Tactical
Strategic
Operational
Tactical
Strategic
Level Operational
Level
Tactical
Level
Operating
Environment
Strategic
Level Operational
Level
Tactical
Level
Tactical
Intelligence
Operational
Intelligence
Strategic
Intelligence
Intelligence
Support
Tactical
Intelligence
Operational
Intelligence
Strategic
Intelligence
OperatingEnvironment
IntelligenceSupport
StrategicLevel Operational
Level
TacticalLevel
TacticalIntelligence
OperationalIntelligence
StrategicIntelligence
OperatingEnvironment
IntelligenceSupport
StrategicLevel Operational
Level
TacticalLevel
TacticalIntelligence
OperationalIntelligence
StrategicIntelligence
Strategic
Level Operational
Level
Tactical
Level
Operating
Environment
Operating
Environment
Strategic
Level Operational
Level
Tactical
Level
Tactical
Intelligence
Operational
Intelligence
Strategic
Intelligence
Intelligence
Support
Tactical
Intelligence
Operational
Intelligence
Strategic
Intelligence
OperatingEnvironment
IntelligenceSupport
StrategicLevel Operational
Level
TacticalLevel
TacticalIntelligence
OperationalIntelligence
StrategicIntelligence
OperatingEnvironment
IntelligenceSupport
StrategicLevel Operational
Level
TacticalLevel
TacticalIntelligence
OperationalIntelligence
StrategicIntelligence
Page 12 of 113
Categories of Intelligence
Similarly, intelligence analysis can generally be divided into three categories, and while
definitions differ, this aide memoire will use the following:
Basic Intelligence – is intelligence, on any subject, which may be used as reference material
in planning. It provides background material on potential adversaries and/or operational
environments, and can deal with equipment, capabilities, personalities, infrastructure, socio-
political and cultural matters, or geography and other environmental information. It can be
permanent, or subject to change, it is normally independent of a specific crisis, and is often
maintained in databases.
Current Intelligence – is intelligence that reflects the current situation at either the strategic,
operational, or tactical level. It almost certainly describes current events and their
consequences, but also may predict changes and estimate future intent. Current intelligence
is perishable and may change rapidly.
Estimative Intelligence – is intelligence that looks to the future in an attempt to prevent
strategic or tactical surprise. Warning intelligence is a form of estimative intelligence.
Types of Strategic Analysis Products
Context analysis or environmental scanning is a way to analyze the environment in which an
intelligence client or decision maker operates. It deals with both internal and external issues and
while it often concentrates on the macro aspects of the situation (the drivers or factors), it may
also be very focused on important details.
Warning analysis provides senior decision makers with effective warning against a range of
complex regional and global actors and events, the adverse outcome of which could threaten
national and Allied interests.
Opportunity analysis is the strategy of assessing the potential for a change to affect an outcome,
relationship, advantage, etc. for either an adversary or a client, and the consequence of that
change.
A strategy of providing clients with assessments that evaluate both the opportunities and the risks
of various options they might be considering will always be valued. The analyst’s role is to
identify the forces and factors that may be influential in the situation and to focus on how to
mitigate bad scenarios from unfolding and how to enhance the prospects for positive
developments to occur.
With any opportunity analysis, three additional key questions should be addressed. First, what
are the benefits of implementing this policy, change or enhancement? Next, what adverse effects
are likely to occur when the implementation takes place? Finally, how will the implementation
affect the overall operation, relationship, etc?
Page 13 of 113
Analytic Rigour
“Analytic rigour” is a term that is widely used, but few analysts or managers can actually
describe what it entails – not very helpful!
For the purposes of this Aide Memoire, analytic rigour is based on the CFINTCOM Analytic
Product Standards (page 99), the CFINTCOM-approved Probability Terms (page 103), the
concept of Analytic Confidence (page 103), the Data Diagnostic process (page 69), which
includes Paul and Elder’s template for “Analysing the Logic of an Article” (page 35), and the
Denial and Deception checklists (page 37); and the Key Assumptions Check (page 70).
Summary:
To exhibit analytic rigour, intelligence analysts should:
- Make accurate judgements,
- Be clear,
- Be insightful, timely and relevant, and
- Highlight trends over time
This should be demonstrated through:
- Identifying confidence in analytic judgements,
- Identifying assumptions,
- Considering alternate hypotheses,
- Identifying indicators, and
- Applying structured analytic techniques.
Finally, analytic rigour depends on the quality and reliability of the evidence as examined
through the lens of the provider, the information itself, its relevance, and the potential for
denial and deception.
See page 113 for a graphic of Analytic Rigour, developed using CmapTools5 software.
Temporary Postmodernism6
Good intelligence analysts, in the process of preparing an analytic product, should make a point
of (within reason) challenging everything. One way of doing this is by temporarily becoming a
postmodernist. From this perspective, the analyst will adopt the position of someone who
believes:
- that there is no privileged access to the truth (as some religious leaders or
environmental advocates would have us think);
- that knowledge is socially constructed (and as societies change, so does knowledge);
- that there are no clear standards for ‘progress’ (a more technologically-assisted or
economically-advanced or culturally-diverse society is not necessarily a better one);
- that no culture is inherently superior; and finally
- that our perspective influences our reality (think about five blind guys holding onto
various parts of an elephant that they are experiencing for the first time).
5 Developed by the Institute for Human and Machine Cognition.
6 Developed from a talk given by Dr Jeffrey Tang of JMU to the 5-Eyes Analytic Training Workshop - 19 Nov 2010
Page 14 of 113
Reflect on the Problem, Determining Possible
Approaches.
What is the Question?
In virtually all aspects of intelligence analysis, the first, and often most perplexing problem is
determining the question that needs to be answered. Whether the analyst is defining a scientific
research question, dealing with a current issue, or supporting an operation and/or investigation,
the importance of this formative aspect cannot be over-estimated.
W5H & SW.
It is perhaps intuitive that the oft-cited W5H can be helpful here, and analysts
are encouraged to answer as many of these questions as possible in defining
the intelligence problem. W5H is, however, insufficient in that it does not
deal with the issue of SO WHAT? Indeed, without an identifiable SW, any
piece of intelligence analysis, no matter how insightful, or clear, may simply
be busywork.
Often the details of what needs to be produced can be found in departmental
policy or standard operating procedures, but there will be times when the
analysis is in response to a client request, and others where the analyst acts as
initiator.
When responding to a client’s request, it is important that the actual client’s needs are identified,
in detail, to the analyst. Given that there are often managers and/or client relations personnel
between the client and the analyst, it is important that everyone involved in the process is attuned
to this requirement. Often, the best way to ensure the analyst understands the client’s
requirements is for the two of them to engage in a discussion of the issue at hand.
When considering an analyst-initiated analysis project, the analyst must be prepared to “sell” the
benefit of the initiative. To this end, the analyst should define the issue, the likely consequences,
and the potential impact on operations, policy and planning.
The questions, who, what, where, when, why, how, and so what, have a broad application in the
processes of intelligence analysis. The table below shows how these seven questions can be
applied when thinking about:
the client of the analysis (and to the client of the client of the analysis),
the content of the analytic product,
the contacts (or sources) that the analyst is relying upon, and
the individuals (or organisations) with whom the analyst should collaborate when
undertaking any analysis.
W5H & SW
Who?
What?
When?
Where?
Why?
How?
So what?
Page 15 of 113
(W5H & SW)C47
Client Content Contact Collaboration
Who is the client? Who is it about? Who? Who is this? Who has extra
information?
What is wanted? What is it about? What? What is being
offered?
What might be
added?
Where is he
located?
Where did/will it
happen? Where?
Where is the info
coming from?
Where are they
located?
When does she
need it?
When did/will it
happen? When?
When was the info
acquired?
When can I reach
them?
Why is it wanted? Why did/will it
take place? Why?
Why is it being
proffered? Why not try?
How can it be
delivered?
How did/will it
come about? How?
How was it made
available?
How can I reach
them?
What is the
implication?
What is the
implication? So What?
What is the
implication?
What is the
implication?
Policy Prescription.
The old adage that intelligence analysis should be policy/operationally relevant but
policy/operationally neutral is extremely important and must be kept in mind throughout the
process. The role of intelligence analysis is to inform operations and policy development, to
make tasks of decision making easier. Intelligence assessments should not tell the clients what to
do; though judgements concerning the consequences of courses of action may be offered.
What is the nature of the question?
Puzzles, Mysteries and Messes.
Analytic problems run the gamut from simple to extremely complicated. A puzzle is the
simplest. A puzzle has clearly defined boundaries, and only one correct answer. An example of
a puzzle is: “Who put the bomb on Air India flight 182?". There is one right answer, and that
answer is at least theoretically knowable even if it is difficult or impossible to prove in court.
A problem is more complex than a puzzle. A mystery has clear boundaries, but has no single
correct answer. An example of a problem in this sense is: “How do we bring the perpetrators of
the Air India bombing to justice?” This is a problem for several reasons, one of which is that
different individuals and legal systems have different opinions and rules on what constitutes
justice in a case of ideologically-motivated mass murder.
7 Developed from a presentation given by Dennis St. John, NSA, at the 4-Eyes Analytic Training Workshop, hosted
by James Madison University’s Institute for National Security Analysis on 1 Sep 2009.
Page 16 of 113
1
One
System
requires
evidence
& reasoning
within a
system
a correct
answer
knowledge
2
No
System
calls for
stating a
subjective
preference
a subjective
opinion
cannot be
assessed
3
Multi
System
requires
evidence
& reasoning
within multiple
systems
better & worse
answers
judgement
1
One
System
requires
evidence
& reasoning
within a
system
a correct
answer
knowledge
1
One
System
requires
evidence
& reasoning
within a
system
a correct
answer
knowledge
2
No
System
calls for
stating a
subjective
preference
a subjective
opinion
cannot be
assessed
2
No
System
calls for
stating a
subjective
preference
a subjective
opinion
cannot be
assessed
3
Multi
System
requires
evidence
& reasoning
within multiple
systems
better & worse
answers
judgement
3
Multi
System
requires
evidence
& reasoning
within multiple
systems
better & worse
answers
judgement
A mess is the most complex of the three, and is also known as a “wicked problem.” A mess is a
complex issue without defined boundaries, and may even lack any clear formulation. Messes
tend to be interactive open systems, unquantifiable, and to contain multiple interrelated
uncertainties. Messes do not have single, correct solutions, and they do not even provide any
way of knowing when a solution has been reached. Instead they have a range of approaches
which may improve the situation, but which carry with them both intended and unintended
consequences which change the nature of the mess. An example of a mess is: “How do we
prevent terrorism?”
Three Kinds of Questions.8
Another way of approaching a problem is to
figure out what type of question it involves. Is
it a question with one definitive answer? Is it a
question that calls for a subjective choice? Or
does the question require the consideration of
competing points of view?
Paul and Elder provide the diagram on the right
as a way of demonstrating these three options.
How much detail?
When determining the nature of the question, it
is important to consider the amount of detail
that will be required to adequately deal with the
issue. There is little sense in expending the
time and energy to produce an all-
encompassing paper on an issue when the only
thing that is required is a simple yes or no answer. Likewise, a short answer may be completely
inadequate in other circumstances.
Time.
An insightful, detailed assessment provided after its usefulness has expired is a waste of time.
Determine when the client requires the assessment and then meet that timing!
Client Knowledge.
Another consideration is the level of knowledge of the client. Again, this is a spectrum where on
one end an expert in the issue only needs to be provided with the specific answer to the question
at hand. This type of decision maker often has a preconceived idea of what the answer “should”
be. The challenge for the analyst, if the answer is in fact something else, will be to convince this
expert that an alternative explanation is appropriate. At the other end of the spectrum is a client
that does not even know that the issue exists. For this latter group of intelligence users, the
8 From The Miniature Guide to Critical Thinking Concepts and Tools, Paul and Elder, p.18
Page 17 of 113
analyst must define the issue in terms the client can understand (unfamiliar words and concepts
will need to be explained), take special care that the logic of the assessment is transparent, and,
throughout, keep in mind that the presentation will need to be compelling enough to hold a
potentially wavering attention.
Issue Redefinition9
Many analytic projects start with an issue statement outlining: what is the issue, why is it an
issue, and how will it be addressed. Unfortunately, it is often the case that the “issue definition”
stage has been given short shrift by either the client (who may or may not know what is actually
wanted) or the analytic team’s client relations group. All this to say, the “issue” that is presented
to the analyst is often not the question that needs to be answered.
To deal with this shortcoming, analysts or analytic teams may want to experiment with different
ways to redefine an issue. This is important, because seemingly small differences in how an
issue is defined can have significant effects on the direction of the research and consequent
analytic products.
Reframing the Question is a structured analytic technique (see page 68) which can be used to get
the analytic process started. While it will not guarantee the analyst pursues the right avenue of
inquiry, it may forestall unnecessary effort on a poorly stated issue.
Identifying Factors
Once the question has been defined, analysts should then undertake the identification of the
factors, criteria and drivers that can be expected to be helpful in understanding the issue - from
the perspective that will be useful to the client.
This identification process can be undertaken through the use of the structured analytic
techniques in the Develop group. The Environmental Scanning techniques – SWOT, Activity
Systems Model, STEMPLES and Understanding Groups (see page 64), and Outside-In Thinking
technique (see page 65) can be especially effective. A quick session with the client can also be
invaluable at this stage.
Summary
The process of reflecting on the problem and determining possible approaches is fundamental to
the tradecraft of analysis. Without a specific and thorough knowledge of the question that needs
to be answered, the type of problem being addressed, the amount of detail that is required, and
the client familiarity with the issue, much analytic time and effort will undoubtedly be wasted.
Concomitantly, and perhaps more alarming, without this prior knowledge, the probability that
the analyst will provide an answer that is inappropriate, incomplete, or unhelpful will increase;
perhaps irrevocably damaging the client’s confidence in intelligence.
9 Based on Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for Intelligence
Analysts, copyright 2011 by CQ Press.
Page 18 of 113
Be Resourceful and Systematic When Collecting
Information, Documenting Sources and Noting
Caveats on Usage.
Once the analyst has determined the question that needs to be answered, the type of problem that
it is, and the detail and timeframe in which it needs to be answered, the next phase of the process,
collection, can begin.
Here, the human condition predisposes us to develop an explanation based on previous
experience and then set out to find evidence that supports this hypothesis. This is to be resisted
as vigorously as possible.
Indeed, during this portion of the analytic process, the objective is simply to gather
data/evidence/information which may be relevant to the question at hand – even if it appears to
be contrary to an initial belief. This should be done as thoroughly and as broadly as possible –
keeping in mind the timeframe/complexity of the task – and the actual question.
Guiding Factors
Analysts undertaking work on a new subject should never be content to rely on the information
that is pushed to them by others, and they should probably not attempt to assimilate everything
that is available on the issue. While analysts will often be constrained by the unavailability of
information, due to the secrecy surrounding much of intelligence work, more often than not, they
will be faced by an overwhelming overabundance of data – much of which is irrelevant to the
task at hand.
One of the best ways of triaging the available information, and determining what type of
information should examined, is by focusing on the data that is related to the factors that have
been identified as likely to provide insight into the problem being studied. That will allow the
analyst to set aside the other, unhelpful but often very interesting, data. This in turn will
facilitate quicker understanding of the issue, in terms that will be of interest to the client.
Often, analysts will be well served by looking outside of their normal environment to collect the
data that is relevant to the issue at hand.
INT Comparison Chart: Pros and Cons10
It is generally accepted that not all sources of information are created equal. Given scarce
resources (especially time) it is often important that the main collection effort be directed against
the sources of highest potential value. To this end, the following chart contrasts and compares
intelligence garnered from open sources with that which is acquired from human, and technical
(signals and imagery) sources.
10
Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analysis Learning Program.
Page 19 of 113
OSINT
Collection risk: low
Cost: low
Pro
broadest coverage
good for identifying emerging issues (can be very up-to-date)
easy to access
generally, minimal risks in collection
cheap
no restrictions on use (easy to share) quality varies, but good for a “first pass”
Con
high volume
unstructured
may alert a sophisticated target
may contain diatribes, polemics, hyperbole, propaganda, and some nuts!
difficult to assess quality (scholarly journals to personal blogs)
language issues
HUMINT
Collection risk: high
Cost: medium
Pro
first-hand information
richest and most detailed information
goes to motives and intent
target will not usually know information reported
flexible - can cover anything (in theory)
only possible source for some hard targets
relatively inexpensive
Con
difficult to recruit good sources on hard targets
may take a long time to set-up a source
policy may limit who can be recruited
sources have biases and personal agendas
they make mistakes, exaggerate their role or access (which may be very limited or
indirect) and they embellish information
money does not guarantee loyalty
a source may be a fabricator, double agent, or dupe
information is hard to verify (i.e. “single and sensitive source”)
good tradecraft is necessary to cope with these problems
large administrative burden (for organization and handler), medium to high personal
risk
Page 20 of 113
SIGINT
Collection risk: low*
Cost: high
*Note: While the personal risk is low, SIGINT collection is politically sensitive and therefore a
high risk for governments.
Pro
large volume of high quality information
remote access and usually not dangerous to collect
broad coverage of potential targets
potential to provide insight into plans and intentions
Con
hard to use or share due to high classification
passive (even random)
high volumes of low yield intelligence makes gold hard to find
costly - requires intensive resources equipment, code-breaking, decoding, translating
and disseminating
many insignificant and significant actors who are poorly informed
many key targets know how to avoid detection
deception easy for knowledgeable targets
low tech targets with no SIGINT profile may be most important challenge to store
and retrieve
IMINT
Collection risk: low
Cost: high
Pro
persuasive
very broad coverage
provides essential information on physical items
intensive magnification from satellites provides very specific information, especially
on military assets, infrastructure, damage assessment, activities which leave marks
good for change detection
photo interpretation is a well-developed art
adversary probably underestimates our capability
Con
not always available
more useful for capability than for intent
more useful for tactical/military than strategic/civilian
material can be misinterpreted
many targets can disguise or hide assets
Page 21 of 113
Collection Planning
The collection process, especially one preparing for a complicated or long-term project, can
often benefit from the assembly of a collection plan. This can be a formalised process like
DND’s CCIRM (Collection Coordination and Intelligence Requirements Management) or a
much simpler, tailored-to-the-task,
spreadsheet or matrix. In either case, the
collection plan will record, in a single place,
what the analysts need, and where they think
they can get it.
Requirements.
Requirement is the term that is used – by
some agencies – to describe any number of
aspects of “what needs to be done” at various
stages in the intelligence process. This lack
of specificity can be confusing (a client’s
requirement is a piece of intelligence advice, a
collector’s requirement is a piece of information, an analyst’s requirement may be an analytic
product, etc.), but the term is helpful nonetheless.
From the analyst’s perspective, an
Intelligence Requirement (IR) is probably
best understood as a question that describes
a gap in intelligence knowledge, while a
Priority Intelligence Requirement (PIR),
as its name suggests, is a gap in intelligence
knowledge that should be filled on a priority
basis. Often Priority Intelligence
Requirements encompass numerous
Intelligence Requirements such as is shown
in the box at the right.
PIRs and IRs such as those shown in the box, however, are often far too broad to be of much use
to a collector or researcher. (Can you imagine asking an imagery analyst to answer the IR –
How much uranium is Lilliput enriching?) For the purposes of identifying exactly what it is the
analyst needs to know, Essential Elements of Information (EEI) and Indicators may be
identified. Using the Lilliput strategic threat example, a reasonable EEI for an imagery analyst
may be:
What is the metric volume of tank six at power-plant Alpha, located at 102.22N / 44.33W?
Similarly, a measurable indicator may be:
A heat-bloom of more than 4,000C at nuclear plant Bravo located at 102 º 22’ N / 40º 33’ W.
When considering the Client Requirement –
What is the strategic threat from Lilliput?
The Analysts PIR may be:
What is Lilliput’s nuclear weapons
capability?
While IRs may be:
How much uranium is Lilliput enriching?
Does Lilliput have a strategic weapons
delivery platform?
Collection Coordination and Intelligence
Requirements Management
DND’s CCIRM is not only a way for members of
the Intelligence Community to coordinate
requirements. CCIRM is also a tracking
mechanism that ensures producers are
accountable for the collection and/or processing
they are tasked with. Even more importantly,
CCIRM tracking serves as the correspondence-
of-record for this activity, upon which senior
leadership can base decisions about
prioritization, resource allocation, and policy.
Page 22 of 113
Many other terminologies can be used to describe exactly the same things as PIRs IRs, EEIs and
indicators. Indeed the Criminal Intelligence Collection Plan, on page 24, uses “Objective” and
“Investigative Questions.”
Indicators.11
The main function of an indicator is to help identify persons, activities, developments, or trends
of interest. Indicators can be used to:
Identify which scenario or alternative future is emerging.
Alert one to unanticipated developments that might otherwise go undetected.
Validate existing hypotheses or viewpoints.
Make the warning process more rigorous.
A pre-established set of indicators are often used to:
Suggest a target’s activities or behaviour is consistent with an established pattern
(backward-looking).
Suggest a given hypothesis is correct or a predicted scenario is emerging (forward-
looking).
The best indicators satisfy all five of the following characteristics. (The first three are critical for
any good indicator, the fourth and fifth are not always possible to satisfy.)
Observable/Collectible/Practical. An indicator must be able to be observed and collected
at suitable time periods.
Valid. An indicator must accurately measure the concept being considered.
Reliable. Data collection must be reliable and different people must be able to see the
same thing.
Stable. An indicator should be useful over time to allow comparisons.
Unique. An indicator should only measure one thing.
Indicators of Change.
In this process, an analyst or team creates a list of indicators or signposts of observable events
that could be expected to become apparent if a postulated situation is developing. The technique
(see page 72) can be used whenever an analyst needs to track an event over time or monitor and
evaluate changes.
11
Adapted from Handbook of Analytic Tools & Techniques, Pherson, 2008, p 16
Page 23 of 113
Indicators Validator.
Indicators, unfortunately, often do not demonstrate all of the desired characteristics as well as the
analyst would like. Indeed, a critical question that is often left un-asked is whether a given
indicator would appear only in the scenario to which it is assigned, or whether it might also
appear in one or more alternative scenarios. The structured analytic technique Indicator
Validator (see page 73) is specifically designed to expose the diagnostic value of indicators in an
attempt to ensure that they are not relied upon inappropriately.
Collection Plans.
Collection plans vary widely in form, format and content, though most will contain some or all
of the following:
Reference to a specific intelligence problem.
Increasingly granular questions that need to be answered.
A list of agencies/sources from which the information can be sought.
A way of indicating which agencies/sources have been tasked/asked/queried on each
specific question.
A way of indicating the status of the requests.
Follow-up action required.
Below is a pair of examples of Collection Plans,12
each dealing with a different strategic security
issue, and each using different terminology to describe the information being sought.
12
Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analysis Learning Program.
Scenario 1 Indicator
Scenario 2
Scenario 3
Scenario 4
Indicator
Indicator
Indicator
Indicators
ValidatorMost
Discriminating Indicators
(Listed in rank order
and by
scenario.)
(Evaluates the
likelihood of each indicator to emerge in each scenario.)
- Highly Likely- Likely- Possible- Unlikely- Highly Unlikely
Scenario 1 Indicator
Scenario 2
Scenario 3
Scenario 4
Indicator
Indicator
Indicator
Indicators
ValidatorMost
Discriminating Indicators
(Listed in rank order
and by
scenario.)
(Evaluates the
likelihood of each indicator to emerge in each scenario.)
- Highly Likely- Likely- Possible- Unlikely- Highly Unlikely
Scenario 1 Indicator
Scenario 2
Scenario 3
Scenario 4
Indicator
Indicator
Indicator
Indicators
ValidatorMost
Discriminating Indicators
(Listed in rank order
and by
scenario.)
(Evaluates the
likelihood of each indicator to emerge in each scenario.)
- Highly Likely- Likely- Possible- Unlikely- Highly Unlikely
Scenario 1 Indicator
Scenario 2
Scenario 3
Scenario 4
Indicator
Indicator
Indicator
Indicators
ValidatorMost
Discriminating Indicators
(Listed in rank order
and by
scenario.)
(Evaluates the
likelihood of each indicator to emerge in each scenario.)
- Highly Likely- Likely- Possible- Unlikely- Highly Unlikely
Page 24 of 113
Criminal Intelligence Collection Plan Objective: COULD CANADIAN RADIOISOTOPES BE USED TO MAKE A
"DIRTY" BOMB
Investigative
Questions
Information Sources Collection
Method
Collection
Method Details 1. What
radioisotopes could
be used in a dirty
bomb?
2 What
radioisotopes are
present in Canada?
3. What level of
interest exists in
terrorist groups to
use a dirty bomb?
4. How accessible
are radioisotopes
along their supply
chain/life cycle?
A) Internal Databases
a) NCDB (3, 4)
b) PROS (3, 4)
c) SCIS (1, 2, 3, 4)
d) etc
B) Open Sources
a) Internet (1, 2, 3, 4)
b) Newspapers (1, 2, 3,
4)
c) Conf. mat. (1, 2, 3, 4)
d) Sci / Tech lit. (1, 2, 4)
C) Dom. Depts / Agencies
a) Health Canada (1, 2)
b) Transport Canada (2,
4)
c) ITAC (1, 2, 3, 4)
d) etc
D) Provincial / Local
Agencies
a) Law enfor. (3, 4)
b) Emerg. Mgmt. (1,2, 4)
c) Min. Environment (2,
4)
E) Foreign Depts / Agencies
a) DHS (1, 2, 3, 4)
b) CIA (1, 2, 3, 4)
c) FBI (1, 2, 3, 4)
d) FEMA (1, 2, 3, 4)
F) Etc.
A) a-g: Intelex &
direct query
B) Intelex
C) Liaison Officers &
direct contact. in-
person and by
telephone. interviews
and questionnaire).
D) Liaison Officers &
direct contact
E) Liaison Officers &
direct contact
A)f) May require
travel to BC. $3,000
B) May need to
purchase reports, buy
subscription, or get
special access. $500
C) Information
sharing issues (non-
derivative use); may
require legal advice.
Possible travel (site
visits). $1,000
D) Information
sharing issues.
Possible travel (site
visits). $5,000
E) Information
sharing issues.
Possible travel (site
visits). $10,000
Page 25 of 113
MSOC East Information Collection Plan Date Created: 20 Dec 06
Intelligence Problem: What is the threat to Canadian Vessels transiting the Suez Canal, Red
Sea and Gulf of Aden Region
Classification: Unclassified
Information Required NLT: 04/01/07
PIR IR
Sources and
Agencies Results Status
OS
INT
IMB
ON
I
TR
AN
SC
OM
ITO
CG
CS
E
AS
TJ
IC
Completed
Pending
Incomplete
PIR 001:
What is the
terrorist
threat within
the specified
region?
IR 001.01 -
What are the
known terrorist
factions within
the region?
X X X X
1.ONI - Reports rec'd.
2. TRANSCOM - Nil
report
3. ITOCG - 3 x associated
reports
rec'd
4. ASTJIC - Regional TRA
rec'd
IR 001.02 -
What have been
the known or
suspected
terrorist
associated
events in this
region?
X X X X X
1. OSINT – Searched,
multiple
results
2. ONI - Maritime reports
rec'd -
(strong assessment)
3. TRANSCOM - Nil
report
4. ITOCG - Included IR
001.01
5. ASTJIC - Included IR
001.01
GAP - Who are
potential emerging
factions in light of the
expanding insurgent
activity in Iraq?
GAP - Will they exploit
this region, and / or
target it?
IR 001.03 -
What have been
the regional
terrorist targets
for attack?
X X X X
1. OSINT - Searched with
multiple
results
2. ONI - Reps rec'd
3. ITOCG - Included IR
001.01
4. ASTJIC - Included IR
001.01
GAP - What have been,
if any, the specific
maritime targets?
ONI - forwarding
additional synopsis
Conclusion pending
Page 26 of 113
Data Sorting13
Data Sorting is particularly effective during the initial data gathering stage when information
elements can be broken out into categories or subcategories for compilation and comparison.
These groupings should be aligned with the factors that were identified during the earlier Identify
Factors (see page 17) phase. The technique is most helpful when massive stores of data can be
assembled in a software program, such as a spreadsheet, where, by grouping material under the
appropriate factor, trends, similarities, differences and gaps can be identified.
The method consists of five steps:
Step 1: Review the factors that have been identified as being important to understanding
the issue at hand and, if appropriate, devise categories and sub-categories into which the
evidence can be broken down. Then place the evidence (data, circumstance, information
or research, or analytic reflection, etc.) into the appropriate portion of the spreadsheet.
Step 2: Review the material in the database or spreadsheet to identify key fields that may
allow you to uncover possible patterns or groupings or gaps.
Step 3: Group those items according to the schema you defined in step 1.
Step 4: Choose a category and sort the data in that category looking for trends or gaps.
Step 5: Review your sorted evidence to see if there are alternative ways to sort it.
Documentation of Sources
In the process of collecting information, it is often important that a record be kept of what has
been found, and where it stored, so that these details may be recalled later. Some analysts can
benefit from very sophisticated computerized document retrieval programs that can handle all of
their information storage and retrieval requirements. Others, however, will need to access a
diverse and mutually exclusive set of information sources (books, TV programs, lectures and
interviews, Internet and classified sources, and personal experience to name but a few). In either
case, tools that will allow the analyst to recollect the data can be of great assistance. To this end,
annotated bibliographies, the Analyst’s Notebook software, and the Cornell Note Taking system
are highly recommended.
Annotated Bibliography.
Annotated bibliographies should, if possible, contain the following items of information:
Source (agency, publication, individual, etc.)
Report number, serial number or message reference number,
Item or article title (abbreviated if necessary),
Date (of publication/storage),
Classification and caveats,
Storage location, and
A short description of the salient points.
13
This section is based on the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic
Techniques, First Edition, March 2008, p 33-35.
Page 27 of 113
Keep track on-line.
Keeping track of on-line sessions can be done using free software such as:
Cogitum Co-citer captures the selected text, its Internet address, its title and the date. You
can also assign your own comments to the texts.
Net Snippets is a sophisticated way to make your Internet research experience more
productive than ever.
Analysts Notebook.
Especially for analysts conducting
investigations in which relationships
between persons, data, events, and
activities are important, the link
diagrams produced by the Analysts
Notebook (or similar software)
provides a convenient method of
assembling and displaying data.
The advantage of using link diagrams
to store data is that they can:
integrate data from different
sources,
highlight key relationships,
serve as a visual briefing aid
for team members, prosecutors
and juries,
show when cases overlap
(when multiple charts are
combined), and
provide a cumulative snapshot of a case.
Examples of annotated bibliography entries.
Heuer Jr, Richard J. Psychology of Intelligence Analysis. Langley: CIA, 1999
http://www.odci.gov/csi/books/19104/index.html (Unclas)
Exceptional & very readable introduction to the issues of analytical bias as it
can apply to cross-cultural situations (as well as analysis in general).
Rotberg, Robert I. When States Fail – Causes and Consequences. Princeton:
Princeton University Press, 2004. ISBN 0691116717 (DG’s Library) (Unclas)
Very good work – intriguing insight into state collapse and failure.
Page 28 of 113
While every analyst will have a personal preference for how to store data, some basic rules can
be useful:
show organizations as boxes and people as circles (icons may also be used),
place people within the box for their organizations,
link individuals and organizations with lines (solid for confirmed links, dotted or dashed
for suspected links),
use coloured links to represent phone calls, financial transactions, commodity flows, etc.,
avoid crossing lines,
place key entities in the centre, and
always include a legend or key.
Additional information can be added to a chart:
grade and source type
o every icon and link can be graded
o the origin of the data can be coded as
source type
use descriptive link types
o links can be generic or indicate the
nature of the association, e.g. owner, subscriber, member, etc.
attributes - visual cues can be added to icons and links
cards
o behind each icon and link, you can create cards to hold additional information
o you can cut and paste to these cards.
o the content is searchable.
The Analysts Notebook is also described on page 76.
Cornell Note Taking System.
Sometimes, information gathering entails attending lectures or
participating in debriefings. If this is the case, the Cornell Note
Taking system provides an easy-to-use, organised, way to
record information.
Note Taking Area: Record the presentation/discussion
as fully and as meaningfully as possible. Use
abbreviations, diagrams and examples.
Cue Column: Soon afterward, sum up the detailed notes
by writing keywords and phrases in the Cue Column.
These concise jottings are clues or “cues” for Reciting,
Reviewing, and Reflecting.
Summaries: Sum up each page of your notes in a
sentence or two. Summarizing clarifies meanings and
relationships, reinforces continuity, and strengthens
memory.
Classification Marking
Alias
Address Postal
BIO Info
DOB : 1972/08/23
Place : Montréal , Quebec
Alias
Address Postal
BIO Info
DOB : 1972/08/23
Place : Montréal , Quebec
C
u
e
s
Note
taking
area
Summaries
C
u
e
s
Note
taking
area
Summaries
Page 29 of 113
In accordance with the Policy on Canadian Government Security (PGS), security classifications
should appear, in capital letters, at the top and bottom of classified documents. Subsequent
dissemination control markings may or may not be capitalised.
In accordance with DND Security Instructions – Ch.27 and the CFINTCOM Intelligence
Production Publication Standard, every title, paragraph, bullet, graph, insert, map, etc. must be
marked as appropriate for that specific portion of the product. And, a document (or a portion
thereof) is to be graded according to its own content (that is the highest element of content that it
contains), and not because of its relationship or reference to another document (or portion
thereof). The consequences of compromise (see the table on the next page) should be used to
help guide the determination classifications, especially when source material is not being directly
referred to (in which case the classification of the source material is to be cited).
While intelligence analysis generally proceeds from an examination and evaluation of
information from classified sources, this, in itself, does not mean that every assessment is
necessarily classified. Similarly, the title of an analytic piece, even though the analysis is highly
classified, needs to be classified based on its own content, and not on its relationship to the
analysis.
There is, however, the distinct possibility that compromise of a compilation of a number of lower
classified elements (paragraphs, titles, pictures, etc.) may increase the degree of injury to the
national interest, because of the compilation; requiring the use of a higher classification. The
complications that this anomaly may produce can be avoided by ensuring that at least one of the
paragraphs (or graphs, or bullets, or inserts, etc.) of the document contains sufficient material to
be of the higher classification in its own right (that is – based on its content). Then, the
document can reflect the classification of the highest element that it contains without any
difficulties.
The listing of product titles emanating from a specific group of analysts during a specific time
period is an example of this compilation effect. Even though all the titles may all be
unclassified, their collocation and identification as emanating from a specific analytic team
during a specific time period would certainly merit protection at the confidential or secret level
as appropriate. In this case, the sentence introducing the list would be classified at the higher
level, while the titles themselves would remain unclassified.
Page 30 of 113
Classifications.
The actual number of
security classifications in
Canada is quite limited.
As shown in the table,
Confidential, Secret, and
Top Secret are only to be
used when dealing with
matters of National
interest. Protected A, B,
and C, on the other hand are reserved for other interests such as protecting personal, business or
other information.
The following table details some of the likely consequences if Top Secret, Secret, or Confidential
material were to be compromised. The examples are not exhaustive.
Consequences of Compromise
Top Secret
Threat to the stability of Canada or friendly nations.
Loss of life.
Exceptionally grave damage to the effectiveness or security of
Canadian and Allied forces.
Exceptionally grave damage to relations with friendly
governments.
Exceptionally grave damage to the effectiveness of extremely
valuable intelligence operations.
Severe long-term damage to the Canadian economy.
Secret
Increased international tension.
Serious damage to international relations.
Serious damage to the operational effectiveness of the Canadian
Forces
Serious damage to valuable intelligence operations.
Significant threats to the national critical infrastructure.
Serious damage to civil order.
Confidential
Damage to Canada’s diplomatic relations.
Damage to the operational effectiveness of the Canadian Forces.
Damage in the short term to economic interests.
Damage to the effectiveness of intelligence operations.
Level Degree of injury to the National Interest
Degree of injury to Other Interests
Top Secret Exceptionally Grave
Secret Grave
Confidential Limited
Protected C Exceptionally Grave
Protected B Grave
Protected A Limited
Page 31 of 113
Control System Marking.14
Control systems are in place within Canada and the 5-eyes nations (Australia, Canada, Great
Britain, New Zealand and USA) to give additional protection to classified information derived
from or concerning sensitive sources, methods, or techniques. The control systems in use within
Canada are:
COMINT – relating to signals intelligence, and
TALENT KEYHOLE – relating to satellite reconnaissance systems and products.
Sub-Control Systems.
Material from or referring to especially sensitive sources and methods may be further
compartmentalised and disseminated to a limited number of recipients on a strict need-to-know
basis. Sub-control system markings include:
GAMMA – provides additional protection for very sensitive COMINT reports, and
ECI (Exceptionally Controlled information) – provides additional protection for very
sensitive SIGINT operations.
Dissemination Control Marking.
Dissemination control markings are used to limit the distribution of material to specific
individuals, groups or nationalities. A dissemination control marking can take any form,
provided it is understood by the reader (for example Task Force Kilo Eyes Only). Dissemination
control markings include:
FVEY – a designation for 5-eyes,
ORCON – a US marking to indicate that dissemination beyond listed addressees is
subject to approval of the originator, and
CEO – Canadian Eyes Only.
Audit Trail
Maintaining an accurate record of the location, author, classification, and caveats restricting the
use of information gathered for the purpose of intelligence analysis can be a daunting task. It is,
however, a vital element in the process and mandatory in many instances, especially when
judicial or inquiry processes are plausible.
Summary
By being resourceful and systematic when collecting information the analyst is in a better
position to make accurate, insightful assessments that will meet the client’s needs.
By documenting sources and noting caveats on usage, the analyst will be better able to recall,
and utilise the material that is found.
14
Developed from the CSE Publication The SIGINT Classification System, OPS-5-15, 22 Aug 2007.
Page 32 of 113
Critically Evaluate the Quality of all Information
Just because a large amount of evidence, properly documented and caveated, has been collected
does not mean that it is all usable. Indeed, much of the data that can be accumulated on any
topic can be wrong, out of context, or deceitful.
Data Triage
When faced with a large
amount of data, and a short
time frame, analysts may
want to do a form of triage
where each report or element
of data is initially judged
relevant or not and then
credible, not credible, or
possibly credible. Having done this classification, the analyst can then deal with the material
that is most likely to contain the needed material.
Data Diagnostics
Diagnosis of the data available can often reveal shortcomings
that, once recognised, can be militated against. To this end,
the analyst should consider the provider of the information,
the quality of the information itself, the relevance of the
information to the problem at hand, and the possibility of
denial or deception.
Provider Diagnostic.15
An evaluation of each provider of information to
determine reliability and credibility will give insight into
the strength and weakness of the material being
provided. Irrespective of the “INT” providing the data,
the analyst should consider, at a minimum, the access
and motivation of both the agency itself, and that of the
agency employee, processor or provider. Analysts can
systematically consider the following questions when
evaluating providers:
15
This section is developed from the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic
Techniques, First Edition, Mar 2008, p 11-13, augmented by material from a Denial and Deception Analysis
Workshop presented 8-9 Apr 2009 by members of the US Foreign Denial and Deception Committee.
Relevant
Not Relevant
Credible Not Credible Possibly Credible
Credible Not Credible Possibly Credible
Data Diagnostics
Provider
Information
Relevance
Denial and Deception
Provider Diagnostic Overview
Agency (HUMINT, SIGINT,
COMINT, OSINT, etc.)
Handler (Agent handler,
editor, translator, processor,
interpreter, etc.)
Access
Motivation
Page 33 of 113
Human Intelligence (HUMINT)
What agency published the report and what is the mandate/political leaning of that
organisation?
Has this agency produced useful material in the past?
Who wrote the report, and what role do they fill in the organisation?
Who controls the source of the information?
How was the source acquired (defector, émigré, agent, walk-in, recruited, etc.)?
How much was paid for the information?
If the report comes from another country – how good is that country at handling
contacts or running agents?
What is the agency’s evaluation of the source within the report?
What reliability does the agency producing the report attach to the information?
Has the reporting officer interjected his or her opinions or assessments?
Is it likely that the handler interpret correctly what the source actually meant (i.e.
were there language, social or expertise barriers)?
Imagery Intelligence (IMINT)
What agency provided the image and what is the mandate/political leaning of that
organisation?
Has this agency produced useful material in the past?
Who wrote the accompanying assessment, and what role do they fill in the
organisation?
Who controls the source of the imagery?
How was the image acquired (satellite, hand held, etc.)?
What does the target know about our imagery sources and capabilities?
Is the collection strategy really able to answer the question being posed:
o Is the frequency of coverage sufficient?
o Is the coverage taking place at the right time of day/year?
o Are the right sensors being used?
What imaging systems (including foreign and commercial) is the target likely
sensitive to and/or guard against?
Communications Intelligence (COMINT)
What agency published the report and what is the mandate/political leaning of that
organisation?
Has this agency produced useful material in the past?
Who wrote the report, and what role do they fill in the organisation?
Who controls the source of the information?
What type of communications circuit is the information coming from (military,
civilian, satellite, air-breathing, etc.)?
Is the circuit enciphered or plain text; was the collection routine or special?
Who translated the conversation, and what organisation do they belong to?
What is the language or transcription proficiency of the translator?
Do the translators understand slang or technical terms associated with the topic?
What does the target know about collection capabilities?
Page 34 of 113
Open-Source Intelligence (OSINT)
What agency published the report and what is the mandate/political leaning of that
organisation?
Has this agency produced useful material in the past?
Who produced the material, and what role do they fill in the organisation?
If based on a foreign language: Who translated the report and what organisation do
they belong to? What is their ability? Do they understand slang? Are they
experienced with this type of problem?
Information Diagnostic.
Having considered the impact of the provider, the analyst should then begin to look at the
information itself. This can be done from two aspects, the quality of the information and the
logic of the information.
Quality of Information.16
Examining the quality of the information independent of the source of the information is
important as critical information can occasionally be found in reports from sources judged to
have low access and a poor record (and vice versa). At a minimum, analysts should ask
themselves the following questions when evaluating the quality of information:
Is the information first-, second-, or third-hand?
Is there information from a separate INT (or source) that corroborates this report?
Is the information consistent or inconsistent with previous information?
Is this a complete transcript (verbatim) or a processed (analysed) summary of the
material?
o Is the report a snippet of a much larger report, conversation, etc?
What was the frequency of collection?
Have there been any recent changes in the frequency of collection?
What was the duration of the collection?
Logic of the Information.
Examining the logic of the information is also very useful and the Paul and Elder template – on
the following page – can be extremely helpful in this regard. While the template is geared
toward analysing the logic of an “article,” the same process can be applied to reporting of almost
any genre.
16
This section is based on the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic
Techniques, First Edition, Mar 2008, p 15-16.
Page 35 of 113
Template for Analysing the Logic of an Article17
1. The main purpose of this article is ________________.
(State as accurately as possible the author’s purpose for writing the article.)
2. The key question that the author is addressing is ______________.
(Figure out the key question in the mind of the author when s/he wrote the article.)
3. The most important information in this article is ________________.
(Figure out the facts, experiences, data the author is using to support her/his conclusions.)
4. The main inferences/conclusions in this article are ___________________.
(Identify the key conclusions the author comes to and presents in the article.)
5. The key concept(s) we need to understand in this article are ______________. By these
concepts the author means ___________________.
(Figure out the most important ideas you would have to understand in order to understand the
author’s line of reasoning.)
6. The main assumption(s) underlying the author’s thinking is (are) _______________.
(Figure out what the author is taking for granted [that might be questioned].)
7a. If we take this line of reasoning seriously, the implications are ________________.
(What consequences are likely to follow if people take the author’s line of reasoning seriously?)
7b. If we fail to take this line of reasoning seriously, the implications are ___________.
(What consequences are likely to follow if people ignore the author’s reasoning?)
8. The main point(s) of view presented in this article is (are) ___________________.
(What is the author looking at, and how is s/he seeing it?)
Relevance Check.
This check, while potentially time consuming, may help ensure that the analyst does not rely on
information that is not relevant to the central issues being assessed. A simplistic approach to
determining relevance is to examine the salience18
of the material. In this regard, the following
questions should be asked:
Does this relate (economically, socially, politically, or militarily, etc.) to the main
intelligence problem?
17
Copied from The Miniature Guide to Critical Thinking Concepts and Tools, Paul and Elder, p.13 18
This section is based on the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic
Techniques, First Edition, Mar 2008, p 17-18.
Page 36 of 113
Does this relate to subordinate issues associated with the main intelligence problem?
Does this make sense with what we know?
Does this make sense with what we think?
Does this beg further questions or highlight changes that need to be addressed
analytically?
Is this consistent with previous information? If not, what caused the change?
Beyond salience, however, relevance can be examined through the lenses of the biases that often
prompt analysts to give credence to material inappropriately. Some of the cognitive pitfalls19
that need to be considered in this regard are:
The vividness criterion – information that is vivid, concrete, and personal has a
greater impact on our thinking than pallid, abstract information that may actually
have substantially greater value as evidence.
The oversensitivity to consistency – in one sense, consistency is clearly an
appropriate guideline for evaluating evidence, but under some circumstances,
consistency can be deceptive. Information may be consistent only because it is highly
correlated or redundant, or because it is drawn from a very small or biased sample.
The bias favouring centralized direction – analysts often overestimate the extent to
which other countries [or groups, or individuals] are pursuing coherent, rational, goal-
maximizing policies, because this makes for more coherent, logical, rational
explanations.
The similarity of cause and effect – heavy things make heavy noises; dainty things
move daintily, etc. is generally true when dealing with physical properties, but there
is little reason for analysts to assume that economic events have primarily economic
causes, that big events have important consequences, or that little events cannot affect
the course of history.
Internal vs. external causes of behaviour – analysts are often inclined to infer that the
behaviour of others is caused by broad personal qualities or dispositions (attitudes,
beliefs, and personality), and expect that these qualities will determine the actor’s
behaviour under other circumstances. Often, not enough weight is assigned to
external causes (incentives and constraints, role requirements, social pressures, etc.).
Illusory correlation – judgements about correlation are fundamental to all intelligence
analysis. For example, assumptions that worsening economic conditions lead to
increased political support for an opposition party, that domestic problems may lead
to foreign adventurism, that …. But, an illusory correlation occurs when people
perceive a relationship that does not in fact exist.
19
The definitions in this section are drawn/quoted from Psychology of Intelligence Analysis, Heuer, 2003, ch 10-11.
Page 37 of 113
Denial and Deception.
An important aspect of evaluating data – whether it comes from HUMINT, SIGINT, Imagery,
MASINT, etc., or you found it on the Internet – is to determine if denial (restricting access to
information) or deception (steering observers away from information) are being used.
An important question for analysts is:
What does the target think our collection/data gathering capabilities are?
To identify and counter denial and deception, analysts should:20
Identify and consider the opponent’s best options, even if there is no current evidence
they are pursuing them.
Beware of obvious, neat patterns that point to one option.
Collect intelligence through as many different and reliable means as possible.
Avoid giving undue importance to a limited number of data points that appear to be
consistent.
Keep an open mind and envision a range of possibilities.
If the possibility of deception is a concern, key reporting can be assessed based on five sets of
criteria:21
Does the
potential deceiver
have Motive,
Opportunity, and
Means (MOM) to
deceive?
Would this
deception be
consistent with
Past Opposition
Practices (POP)?
Do we have
cause for concern
regarding the
Manipulability of
Sources
(MOSES)?
What can we learn from our Evaluation of Evidence (EVE)?
20
These points come from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar
2008, p 20-21. 21
From the CIA Tradecraft Review, A Tradecraft Primer: Structured Analytic Techniques for Improving
Intelligence Analysis, Volume 2 Number 2, Jun 2005, p 14.
Motive, Opportunity and Means
(MOM)
Manipulation of Sources (MOSES)
Motive – What are the deceiver’s goals?
Channels – What means are available?
Risks – What are the risks of discovery?
Costs – Can deception be accomplished?
Feedback – Can deceiver monitor its use?
Is the source reliable?
Does the source have access?
How good are the source’s bona fides?
Is the source vulnerable to control or manipulation by the target?
Past Opposition Practice (POP) Evaluation of Evidence (EVE)
Does the deceiver have a history of
deception?
Does the deception fit past patterns?
If not, are there other historical precedents?
If not, are there changed circumstances that would explain this form of deception?
How accurate is the source’s
reporting?
Is the whole chain of evidence available?
Does evidence from one source conflict with others?
Do other sources of information
provide corroborating evidence?
Is the absence of evidence unusual?
Page 38 of 113
However, it is also possible that the
source is honestly reporting that which
he/she believes to be true. In that case,
analysts need to ask:
Is the Source of the
information simply Naïve
(SON)?
Analysts should be most concerned
about the possibility of deception in the
following situations:22
The analysis hinges on a single key piece of information;
The potential deceiver has a history of being deceptive;
Data is received at a critical time when those involved have a great deal to gain/lose.
Accepting the new information would require the decision maker to expend or divert
significant resources;
Accepting the new information would cause the analyst to alter a key assumption or
key judgement;
The adversary or the competitor could track its adversary’s behaviour and decision-
making process through an established feedback channel.
Data Quality Diagnostic23
A Data Quality Diagnostic is one way to make analytical arguments or intelligence gaps more
apparent and is useful because it prompts a focus on four areas, specifically, the provider of the
evidence, the information provided, its relevance, and the possibility the evidence is part of a
denial and deception effort. Although a quality diagnostic can not ensure the accuracy of
analytic judgements, its use usually enhances the credibility and usability of intelligence
assessments.
When you conduct a diagnostic review of evidence, use the matrix below to compile your
findings. Rate your confidence in the evidence as High, Medium, Low, or Questionable. For
Denial and Deception, rate the likelihood of deception occurring.
22
Copied from Critical Thinking for Strategic Intelligence, Pherson and Pherson, 2013, p 105. 23
Developed from a USEUCOM CTSA Course Diagnostic Exercise, Mar 2008
Source Naivety (SON)
Is it just that the well meaning source is naïve?
Might the source be attempting to influence operational
plans or policy to advance a personal agenda?
Is the source evangelically fervent about the subject under
discussion?
Does the source subscribe to “non-mainstream” beliefs?
Might the source simply be trying to please you or the
interlocutor?
3.
2.
1.
Deceptio
n
Rele
va
nt
Da
ta
Pro
vid
er
Overa
ll Ratin
g
Diagnostics
Data Quality Diagnostic
Page 39 of 113
Propaganda
Propaganda is simply another form of denial and deception.
How to identify propaganda - Red Flags24
The Publication
1. Known for extremist views.
The Source
2. An obscure “institute” or academic and likely a mouthpiece for a special interest group.
Where do they get their funding? Who are the directors? What are their backgrounds?
The Content
3. Strong emotional aspect
a) Ominous, stirring or patriotic music and images
b) Associates a person, event or idea with something hated or feared, e.g. Nazis
c) Use of slogans, e.g. "blood for oil" , "cut and run" , “united we stand”
d) Use of virtue words, e.g. Peace, happiness, security, wise leadership, freedom, liberty …
4. Poor reasoning
a) Illogical (or non-intuitive) relationships between concepts
b) Sweeping conclusions from mere anecdotal evidence
c) Issue framed to favour one point of view
d) Irrelevant or questionable data
e) Vague, undefined terms
5. Misrepresentation - false or missing information (half of the story)
6. Oversimplification
a) Simple answers to complex social and political questions
b) Blame assigned to an individual or group (i.e. scapegoating).
c) Misleading stereotypes or labels.
d) Blanket statements
7. Persuasive Aim
a) cites or associates prominent figures to a position idea, argument or action.
b) repeats ideas until they are accepted as truth
c) presents ideas as the view of the majority (so get on the bandwagon)
d) opposition (to author’s premise) would be unpatriotic, undemocratic, inhumane
24
Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analysis Learning Program.
Page 40 of 113
Checklists for Evaluating Information
HUMINT Reporting
Evaluation Status
o New,
o Developing,
o Established
Reporting Record
o On trial,
o No reason to
doubt,
o Partially
corroborated,
o Reliable
Access
o Indirect,
o Direct,
o Quoting,
o Opportunistic,
o Occasional,
o Regular
Four Step - Web Page Evaluation Checklist.25
Next page.
25
Developed by John Pyrik for use in the Canadian interdepartmental Intelligence Analysis Learning Program.
Alpha-Numeric Source-Data Rating
Reliability of Source Credibility of the
Information
A Completely reliable 1 Confirmed by other sources
B Usually reliable 2 Probably true
C Fairly reliable 3 Possibly true
D Not usually reliable 4 Doubtful
E Unreliable 5 Improbable
F Reliability cannot be
judged 6 Truth cannot be judged
Page 41 of 113
Name of Website: Assessed Reliability
Low 1- 2- 3 -4- 5 High URL / Address: http://
1. Type Advocacy Business Info/Ref News Personal Entertainment Meta-tags – Who are they trying to attract to their
website (view / source)?
Older Versions – How did the site evolve?
archive.org)?
2. Content Accuracy - errors of fact or logic
- misspellings, poor grammar
- incorrect dates
Authority - author unqualified, uncited
- poor reputation
- sources undocumented
Objectivity - any blatant bias (terms, etc)?
- persuasive aim?
- single or multiple POV?
- any sponsors or advertising?
Currency
- When was it last updated?
- Any dead links?
Coverage - omissions?
3. Owner / Author: Full legal company name:
-Check copyright and privacy statements.
Who registered the domain?
Who incorporated the company? Officers / directors?
4. Affiliations and Associations? Who do they link to?
Shared premises? Tip: Google phone numbers and
addresses
Who links to them?
- Nature of association
- Affect on credibility
What do others say about them? Google names.
Consistent with similar sites?
Date Completed
Analyst / Investigator
Four Step
Web Page Evaluation Checklist
Security Classification
Page 42 of 113
Develop Multiple Hypotheses / Explanations
Formulating Hypothesis
Generating hypotheses26
is at the heart of intelligence analysis, and is one of the main ways in
which analysts try to provide meaning for their data. Analysts should always try to generate
several explanatory hypotheses, to make sure that they have covered a wide range of
possibilities. In intelligence analysis, as in science, hypotheses are preliminary explanations, put
forward for examination and testing, and not a final judgement. This is one of the aspects of
effective analysis in which creativity and imagination are most important.
Black Swan Rule.
No matter how many white swans one finds to prove that all swans are white, it only takes one
black swan to disprove this hypothesis. Disproving a hypothesis is far more emphatic and valid
than trying to prove it.
Value Added.27
Generating multiple hypotheses at the start of a project helps analysts avoid common analytic
pitfalls such as:
Coming to premature closure;
Being overly influenced by first impressions;
Selecting the first answer that appears “good enough” (satisficing);
Focusing on a narrow range of alternatives representing marginal, not radical, change;
Opting for what elicits the most agreement or is desired by the boss;
Selecting the alternative that avoids a previous error or replicates a past success.
Three Basic Approaches.
There are, in the broadest sense, three basic approaches28
to developing hypotheses or
explanations that can be used by intelligence analysts.
The first is Situation Specific. In this case, the issue is seen as unique and the
information/intelligence gathered is dealt with concretely, not as generalisation. As the situation
develops, a single scenario is created that hangs together as a plausible narrative. Newer
information is then compared with the original narrative for consistency, and over time the
picture is increasingly fleshed out until a convincing storyline has been built. This approach, on
its own, risks being dismissive of information that does not fit the evolving picture, possibly
over-emphasising the value of confirmatory information.
26
Quoted from the An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar 2008, p.25 27
Quoted from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.11 28
The three approaches to hypothesis generation is developed from the DIS Principles of Defence Intelligence
Analysis 2nd
Edition - Summer 2006 p 38-39
Page 43 of 113
The second approach is Theory Based, though the theory is often simply a set of commonly
accepted assumptions. While the Situation Specific approach attempts to extract a plausible
narrative from existing/emerging data, the Theory Based approach starts with a narrative and
attempts to fit the data to it. Problematically, any strongly held theory (either science-based or
assumption-based) will tend to resist the proper consideration of divergent information.
The third approach is that of Comparison. Here, the evidence at hand is compared to other
well-understood situations or circumstances. While this is generally useful in the technical
intelligence realm where new data is compared to known systems and capabilities, it becomes
problematic when truly comparable analogies are difficult to identify.
Note that these three approaches are not mutually exclusive.
Types of Hypotheses.
Several types of hypotheses29
can be generated, such as:
Narrow but Deep – These hypotheses will look at the particulars of a specific case, such
as a country or organisation, and attempt to discover causes of a particular situation,
identify key drivers or factors, and speculate on ways the situation may develop. This
approach is most useful when the analyst has substantial data.
Wide but Shallow – In contrast, these hypotheses will consider similar situations in a
number of similar cases. This approach is likely to be especially helpful if the analyst has
only limited data on a particular situation.
Interdisciplinary – Another way to generate hypotheses is to look at the problems from
a different perspective. If the analyst has been thinking about the issue from a political or
military point of view, considering it from the perspective of economics, sociology or
some other discipline may help to provide new insights and possible explanations.
A Good Hypothesis
Is written as a statement, not as a question.
Is testable and falsifiable.
Predicts the anticipated results clearly.
Hypotheses (and Scenario) Generation
The generation of hypotheses or scenarios can be facilitated through the use of the structured
analytic techniques contained in the Develop and Generate Scenarios groupings.
Structured Brainstorming. (See page 62)
Outside-In Thinking. (See page 65)
Hypothesis Generator. (See page 66)
Hypothesis Review. (See page 67)
Quadrant Hypothesis Generation. (See page 82) and Alternatives Futures. (See page 83)
29
Developed from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis, Mar 2008,
p.25
Page 44 of 113
Challenge Assumptions, Mindsets and Biases
The process of perception links people to their environment and is critical to accurate
understanding of the world about us. Accurate intelligence analysis obviously requires accurate
perception. Yet research into human perception demonstrates that the process is beset by many
pitfalls. Moreover, the circumstances under which intelligence analysis is conducted are
precisely the circumstances in which accurate perception tends to be most difficult.30
The limitation of the human mind will impair or cloud the ability of an analyst to assimilate,
interpret, and understand information. Knowing these limitations and guarding against these
tendencies may, however, help analysts produce intelligence that is more complete and objective.
Assumptions
Implication of Assumptions.
As is shown in the box, assumptions are the
fog through which information in interpreted
to arrive at inferences or conclusions. An
assumption is an idea that is taken for granted,
it may be explicit (clearly stated) or implicit
(not stated at all) – IT MAY BE
INCORRECT! It will often take the guise of
common knowledge.
Identifying assumptions can be one of the most difficult challenges that an analyst will face, but
if an influential assumption is incorrect, the likelihood that the resulting inference or conclusion
is correct drops precipitously.
(Information, in this context, is made up of evidence, indicators, factors and/or proof, [Stuff
That Matters] which are all different ways of describing observable phenomena that have
relevance to an analytical problem.)
Key Assumptions Check.31
One of the best ways of mitigating the influence of assumptions is the Key Assumption Check
analytic technique (see page 70).
The goal of the key assumptions check is not to undermine or abandon key assumptions; rather,
it is to make them explicit and identify what information or developments would demand
rethinking them.
30
Quoted from Psychology of Intelligence Analysis, Heuer, 2003, p5 31
From the CIA Tradecraft Review, A Tradecraft Primer: Structured Analytic Techniques for Improving
Intelligence Analysis, Volume 2 Number 2, Jun 2005, pp 7-9
Proof
Evidenc
e
on
Information
The Reasoning Process
Assumptions
Indication
s
Factor
s
Assessment
Factors
Evidence
Inference
Conclusion
Information
The ReasoningProcess
Assumptions
Indications
Proof
Appreciation
Proof
Evidenc
e
on
Information
The Reasoning Process
Assumptions
Indication
s
Factor
s
Assessment
Factors
Evidence
Inference
Conclusion
Information
The ReasoningProcess
Assumptions
Indications
Proof
Appreciation
Page 45 of 113
Mirror-Imaging
(Ethnocentricity)
This is the inability to see the
world through the eyes of a
different national or ethnic group
or the inability to put aside one’s
own cultural attitudes and
imagine the world from the
perspective of those belonging to
a different group. An ethnocentric
perspective is especially
dangerous in the intelligence
context because it can distort
important aspects of strategic
thinking, especially where
problems of perception and
prediction are involved.
Checking for key assumptions requires analysts to consider how their analysis depends on the
validity of certain premises, which they do not routinely question or believe to be in doubt. A
four step process will help analysts:
Step 1: Review what the current analytic line on this issue appears to be; write it down
for all to see.
Step 2: Articulate all the premises, both stated and unstated in finished intelligence,
which are accepted as true for this analytic line to be valid.
Step 3: Challenge each assumption, asking the questions of the Key Assumptions Check
analytic technique. Explore why it “must” be true and whether it remains valid under all
conditions.
Step 4: Refine the list of key assumptions to contain only those that “must be true” to
sustain your analytic line; consider under what conditions or in the face of what
information these assumptions might not hold.
Mindsets and Biases
A substantial body of research in cognitive psychology and decision making is based on the
premise that … cognitive limitations cause people to employ various simplifying strategies and
rules of thumb to ease the burden of mentally processing information to make judgements and
decisions. These simple rules of thumb are often useful in helping us deal with complexity and
ambiguity. Under many circumstances, however, they lead to predictably faulty judgements
known as cognitive biases. Cognitive biases are mental errors caused by our simplified
information processing strategies. It is important to distinguish cognitive biases from other forms
of bias, such as cultural bias, organizational bias, or bias that results from one’s own self-interest.
In other words, a cognitive bias does not result from any emotional or intellectual predisposition
toward a certain judgement, but rather from subconscious mental procedures for processing
information. A cognitive bias is a mental error that is
consistent and predictable.32
Implication of Mindsets.
Mindsets33
are mental models that people develop over time,
based on factors such as education, family environment,
professional experience and travel. Mindsets are easy to form
but difficult to change. Data that does not fit into a given
mindset is often dismissed or devalued. Examples of mindsets
are:
Self-delusion – ambiguous information is interpreted
in favour of what is already believed.
Mirror Imaging – people assume that others, even in a
different culture, would deal with a situation more or
less the same way in which they would.
Groupthink – a small group, with strong leadership,
32
Quoted from Psychology of Intelligence Analysis, Heuer, 2003, p110 33
From the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p 23
Page 46 of 113
tries to reach a decision under pressure, but it discounts alternatives (and therefore is
more likely to make the wrong decision) in the interest of maintaining the solidarity of
the group.
Paradox of Expertise – it is exactly those who have worked intently on an issue for a
long time who are least able to detect potential change; moreover, their experience helps
to make these experts overconfident about the accuracy of their judgements.
Implication of Biases.
Biases34
arise from the structure of the mind, and human beings are all born with mental
frameworks and dispositions that form the basis for eventual biases. Examples include:
Hindsight Bias – looking back on an event, once the outcome is known, and wrongly
believing that events were clearer in nature and easier to predict than they actually were.
Availability Bias – the mind gives greater weight to data that is vivid, concrete, and
personal than to more abstract information such as statistics.
Pattern Bias – the mind attributes a pattern or a controlling force in situations when data
are actually random or coincidental.
Confirmation Bias – we see what we expect to see, and more data is needed to recognise
and understand what has not been expected.
Biases were examined earlier when considering the relevance of evidence. See page 36 for an
explanation of the biases of:
Vividness
Oversensitivity to consistency
Tendency to favour centralised
direction
Similarity of Cause and Effect
Internal vs external causes
Illusory correlation
Countering Mindsets and Biases35
Mindsets and biases are often mitigated by simply implicating an individual or individuals into
the analytic process that do not share the mindset or biases of the original analyst or group. The
Review Pre-Publication and Use Work Groups families of structured analytic techniques are
quite good at this; particularly:
Devil’s Advocacy. (See page 61)
Red Cell. (See page 60)
Team A/Team B (See page 58)
Dialectic Inquiry. (See page 94)
Red Team. (See page 59)
34
From the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p 23 35
Adapted from the GFF An Intelligence Analysis Primer: Six Steps to Better Intelligence Analysis,, Mar 2008, p
24
Self Test:
How long have I held my current views?
Am I dismissive of alternate views?
Do I only read sources that reinforce my
views?
Do I “cherry pick” evidence that fits with my
views while discounting information that
doesn’t?
Page 47 of 113
Prevention of Fixed Mindsets 36
Much work is being done on the cognitive and neuroscience aspects of fixed mindsets and many
researchers are concluding that factors such as Western child-rearing practices, chronic daily
work routines, and the standard American diet, are significant contributing factors. While there
is little we can do about our child-rearing practices, there are steps we can take to avoid the
development of fixed mindsets.
Research is showing that analyst managers can help their analysts avoid the debilitating
effects of fixed neurological structures caused by specialised long-term work routines by
implementing work programs that are analogous to a cross-training physical fitness
regime. That is, analysts should be regularly challenged by very diverse work tasks,
forcing them to constantly use various functions and areas of their brains.
Recent groundbreaking explorations of the connection between exercise and the brain’s
performance show that even moderate exercise will supercharge mental circuits to beat
stress, sharpen thinking, enhance memory, and much more.
The brain in general and its capacity to change and grow in particular is very sensitive to
environmental influences like stress, dehydration, nutrition and diet. To this end, along
with dietary restriction, fasting, and calorie restriction, potentially helpful nutrients so far
examined include:
o Omega-3 fatty acids
o Curcumin (the major component of turmeric), green tea
o Antioxidants: Flavonoids, Vitamin E, Resveratrol
o Ketogenic diet (Coconut Oil, Medium Chain Triglycerides)
o Dietary branched chain amino acids
o Caffeine
36
Developed from a briefing given by Dr Michael Axel of the Berlin Free University to the 5-Eyes Analytic
Training Workshop on 11 Feb 2013.
Page 48 of 113
Build Collaborative Networks
Canadian Security and Intelligence Community
Why Collaborate?37
Collaboration is a fundamental intelligence community concept and an essential element of
analytic tradecraft. By bringing broad-based experience to bear on complex intelligence issues,
clients can be provided with the context, perspective, and implications that these issues demand.
The best intelligence analysis is produced through collaboration.
It is almost impossible to conceive of an intelligence product that can be created solely by a
single analyst or production element. A report detailing development of a new weapons system
must go beyond the technical characteristics of the system and include: a discussion of the
military strategy, force structure, and infrastructure that is available or necessary to support it
use; the direct and indirect effect on deployed or deploying forces; and the political, diplomatic
37
Developed from the draft DIA Tradecraft Note on Collaboration, dated 27 Aug 2010.
Page 49 of 113
and foreign relations impacts of the new system. A product discussing cyber threats falls short if
it does not discuss the implications to the nations, organisations, or personnel potentially affected
by the development. The argument – that the integration of all knowledge pertinent to a given
issue is not an intelligence responsibility – can only result in leaving clients to perform the
integration on their own.
Collaboration is an active and iterative process of partnering, in all phases of the intelligence
process, with others who share an interest in the issue (and indeed, some who do not).
Collaboration distinguishes intelligence analysis from other information professions such as
academic research and journalism and it includes not only other members of the intelligence
community, but outside experts as well. Well done, collaboration:
enriches finished intelligence with multidisciplinary analysis,
reduces the impact of individual bias on the analysis,
helps identify and deal with assumptions,
helps clarify uncertainties,
aids in the identification of influential factors, and
encourages the incorporation of alternative hypotheses.
Absent appropriate collaboration, products will have a higher chance of being incomplete,
confusing, or even contradictory.
Creating a Collaborative Culture38
The creation of a culture of collaboration is neither simple, nor spontaneous. Rather it is a
deliberate process requiring the commitment of both analysts and analyst managers. To this end,
analysts should:
Initiate collaborative relationships with colleagues outside their work unit, directorate,
department/agency, and country, to include other analysts, operators, policy makers,
academics, NGO, PVO, and others.
Utilise collaborative tools to overcome organisational and geographic boundaries.
Credit those who have collaborated on projects and provide appropriate feedback to
individuals outside of the community. People are more likely to collaborate again if they
are aware of their impact.
Concomitantly, analyst managers should:
Organise and lead topically focused forums creating relaxed environments for analysts
and others to meet and form the foundations for communities of interest.
Insist that analysts drafting products solicit input from all stakeholders, especially those
outside the analysts’ work unit, before putting pen to paper.
Hold analysts accountable for initiating and sustaining effective relationships with others
outside their immediate work unit during periodic performance review.
Recognise and reward analysts who lead communities of interest on their issues or
otherwise collaborate effectively with their counterparts.
Develop a process/system of handing-over contacts from one analyst to another during
personnel changes.
38
Developed from the draft DIA Tradecraft Note on Collaboration, dated 27 Aug 2010.
Page 50 of 113
Team/Group Collaboration39
Teams are often used in intelligence analysis with the objective of producing better analysis than
could be done by an individual working alone. Groups or teams often have:
More resources than individuals
Greater diversity of resources
More flexibility in deploying their resources
More opportunities for collective learning
The potential for “synergy”
However, the performance of groups/teams is often subpar when compared to “nominal” groups.
Why is this the case?
Poor leadership – this is the most often cited reason.
A team is used for work that is better performed by an individual.
The task is too hard.
The team is set up in a way that stymies its potential – this may be the most common
reason.
When should a team be used for analytic work? Sometimes there is no real choice, but when
there is, consider the following:
When the desired result is a creative composition – never use a team. The analogy here
is: while it takes a full orchestra to render Tchaikovsky's famous 1812 Overture,
Tchaikovsky himself imagined and then produced the composition.
When confronted by an urgent, focused, multi-disciplinary problem – always use a team.
But when there is room to choose, what type of team is called for?
Models of Collaboration.40
Four models of collaboration worth considering are: Task
Force, Experts Group, Professional Community, and
Interested Parties. To decide which model is appropriate,
two questions need to be asked:
1. Where might the answer(s) lie? In a small or
large group?
2. What degree of coordination and control is
needed? High or Low?
These two questions will result in four combinations of
small/large and high/low.
Small Group + High Control = Task Force.
39
Adapted from a briefing to the GFF COI POI on 24 Feb 2009 by J. Richard Hackman from Harvard University. 40 Developed from an article by John Pyrik published in the Nov 2009 edition of the Intelligence Analyst Training
Newsletter.
Interested
Parties
Experts
Group
Professional
Community
Task
Force
De
gre
e o
f O
rga
nis
atio
n
Lo
w -
----
----
-H
igh
Group Size
Small/Closed --- Large/Open
Interested
Parties
Experts
Group
Professional
Community
Task
Force
Interested
Parties
Experts
Group
Professional
Community
Task
Force
De
gre
e o
f O
rga
nis
atio
n
Lo
w -
----
----
-H
igh
Group Size
Small/Closed --- Large/Open
Page 51 of 113
Here, cooperation is mandatory, activities are centrally directed, and the results
are generally for the benefit of one party.
Small Group + Low Control = Experts Group
In an Experts Group, cooperation is voluntary, activities are determined
collectively, and benefits are shared equally (or at least are accessible to all
parties). Experts groups are often used to collaborate on a product that will be
published by one of the participants.
Large Group + High Control = Professional Community
The Professional Community is generally a large and diverse group with a high
degree of coordination. The process of interaction is often structured and
membership has enduring multi-party benefits.
Large Group + Low Control = Interested Parties
Here, the process of interaction is informal and ad hoc and the benefits accrue
primarily to one party. The boundaries of the group are often blurred and could
involve a potentially unlimited number of self-selected "interested parties"
associated in a loose fashion.
Developing Effective Teams.
The role of the team leader is crucial in the
development of good teams. A good team leader
will:
Decide about the kind of team that is
appropriate for the work to be done.
Get the essential and enabling conditions
in place – and keep them there.
Coach at the margins to help the team take
full advantage of its favourable
performance circumstances.
For more about designing and leading effective teams, see the book Leading Teams or visit
http://www.leadingteams.org. Alternatively, the Team Diagnostic Survey assesses the standing
of a team on the conditions that foster team effectiveness and provides a diagnostic profile of the
team’s strengths and weaknesses. For online access, go to https://research.wjh.harvard.edu/TDS.
Team Formation.41
Creating a team from a divergent group of individuals, be they from disparate sections of the
same organisation, inter-departmental or even international, can be a challenge. One way to help
ensure that all members are in agreement about the team’s main purposes and, more importantly,
are able to bring the full range of their knowledge, skill and experience to bear in accomplishing
these purposes, can be facilitated through the use of a team formation exercise. One such
process is as follows:
41
Developed from the Blue Team Launch Exercise, produced by J. Richard Hackman from Harvard University.
Developing effective teams:
Make the purpose of the team
clear – identify a specific leader.
Identify (and inform the team of)
individual member knowledge,
contacts, and expertise.
Identify additional needs – assign
a team member to attain them.
Establish clear team norms.
Make “course corrections” as
necessary.
Page 52 of 113
Step One: Selection of Timekeeper and Recorder (these roles should be filled by the team
leader if there is one).
Step Two: Individual Work – have each individual identify:
o the main purpose of the team as they understand it,
o the main consequences of the team achieving its purpose, and the team not
achieving its purpose, and
o on a scale of 1 to 5, how challenging it will be for the team to achieve its purpose.
Step Three: Discussion by Pairs – have each pair identify:
o how their separate descriptions of the main purpose agree and disagree,
o the knowledge, skill, experience, perspectives, and relevant outside contacts that
each person brings to the team that could be helpful in achieving the team’s main
purpose (this should be accomplished by interviewing each other), and
o a short list (less than four items) of things they want to report to, or ask of the
remainder of the team.
Step Four: Reconvene, Report and Discuss – the recorder or team leader should facilitate
the discussion of, and summarise and record the following:
o the knowledge, skill, experience, perspectives and relevant outside contacts that
are available to the team,
o the additional expertise that is needed, and who specifically is in the best position
to obtain it from resources outside the team, and
o team norms.
Step 5: Wrap Up – the team should:
o identify anything else that needs to be addressed before work can start,
o if a team leader is not already in place, appoint one, and
o based on what has been discussed and concluded, decide on what the team should
do to get off to a good start with actual work on the task.
Consensus vs Dissent42
Common Pitfalls with Small Groups.
Research suggests that the desire for consensus is an important cause of poor group decisions.
Development of a group consensus is usually perceived as success, but, in reality, it is often
indicative of failure. Premature consensus is one of the more common causes of sub-optimal
group performance. It leads to failure to identify or seriously consider alternatives, failure to
examine the negative aspects of the preferred position, and failure to consider the consequences
that might follow if the preferred position is wrong.
42
Developed and quoted from Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques
for Intelligence Analysts, pp 300-303 & pp 257-260, copyright 2011 by CQ Press.
Page 53 of 113
Other problems that are less obvious, but no less significant have been documented extensively
by academic researchers. It frequently happens that some reasonable satisfactory solution is
proposed that all members can agree with, and the discussion is ended without further search to
see if there may be a better answer. Such a decision often falls short of the optimum that might
be achieved with further inquiry.
A phenomenon known as group polarization leads, in certain predictable circumstances, to a
group decision that is more extreme than the average group member’s view prior to the
discussion. Social loafing is the phenomenon that people working in a group will often expend
less effort than if they were working to accomplish the same task on their own. In any of these
situations, the result is often an inferior product that suffers from a lack of analytic rigour.
Benefitting from Diversity.
Improvement of group performance requires an understanding of these problems and a
conscientious effort to avoid or mitigate them. The literature on small-group performance is
virtually unanimous in emphasizing that groups make better decisions when their members bring
to the table a diverse set of ideas, opinions, and perspectives.
Laboratory experiments have also shown that even a single dissenting opinion, all by itself,
makes a group’s decisions more nuanced and its decision-making process more rigorous. The
research also shows that the benefits from dissenting opinions occur regardless of whether or not
the dissenter is correct. The dissent stimulates a reappraisal of the situation and identification of
options that otherwise would have gone undetected.
To be effective, however, dissent must be genuine, not generated artificially as in some
application of the Devil’s Advocacy technique. It should also be reasonable. If the person
voicing dissenting views is known to the group as a habitual contrarian or maverick, his or her
comments run the risk of being dismissed by the group regardless of merit.
Advocacy vs Objective Inquiry.
Diversity of opinion is, of course, a double-edged sword as it can become a source of conflict
which degrades group effectiveness. Analysts must engage in inquiry, not advocacy, and they
must be critical of ideas but not critical of people.
In a task-oriented team environment or where Devil’s Advocacy, Red Cell, or other such
techniques are in use, analysts adopt the role of advocate which often leads to emotional conflict
and reduced team effectiveness. Advocates tend to examine evidence in a biased manner,
accepting at face value information that seems to confirm their own point of view and subjecting
any contrary evidence to highly critical evaluation. Advocacy is appropriate in the courtroom,
but it is not an appropriate method of discourse within a team, especially when power is
unequally distributed among the participants, when information is unequally distributed, and
when there are no clear rules of engagement – especially about how the final decision will be
made.
Page 54 of 113
On the other hand, an effective mitigation of all of these challenges can be found in a process of
objective inquiry – for example, see the Dialectic Inquiry process outlined on page 94 – which
can lead to new and creative solutions to problems, especially when it occurs in an atmosphere of
civility, collaboration, and common purpose. The table below displays the differences between
advocacy and the objective inquiry expected from a team member or colleague.
Advocacy Inquiry
Concept of decision
making A contest
Collaborative problem
solving
Purpose of discussion Persuasion and lobbying Testing and evaluation
Participants’ role Spokespeople Critical thinkers
Pattern of behaviour Strive to persuade others
Defend your position
Downplay weaknesses
Present balanced arguments
Remain open to alternatives
Accept constructive criticism
Minority views Discouraged or dismissed Cultivated and valued
Outcome Winners and losers Collective ownership
Forcing a Consensus.
Notwithstanding the insistence above that dissenting views are valuable in intelligence analysis,
there are times when a consensus simply must be achieved, or where the strongly held views of
an individual or group is becoming inimical to achieving the task at hand. In these cases, it is
important to be able to manage the confrontation that is almost inevitable. To this end,
adversarial collaboration techniques, based on the forced requirement to understand and address
the other side’s position, rather than simply dismiss it, may be of use. In the techniques that
follow, mutual understanding of the other side’s position is a bridge to productive collaboration:
Key Assumptions Check – Given the difficulty inherent in identifying assumptions, and
their taken-for-granted nature, the Key Assumptions Check (see page 70) can be a very
effective method to help understand what underlies a conflicting judgement. If a Key
Assumptions Check has not already been done, each side can undertake the process and
then share the results with the other. Discussion should then focus on the rationale for
each assumption and suggestions for how the assumptions might be either confirmed or
refuted. The discussion should focus on refuting the other side’s assumptions, rather than
on supporting one’s own.
Analysis of Competing Hypotheses – When opposing sides are dealing with a collegial
difference of opinion, with neither side firmly locked into its position, ACH (see page 74
may provide a good structured format for helping to identify and discuss the differences.
Both parties agree on a set of hypotheses and then rate each item of evidence or
assumption as consistent or inconsistent with each hypothesis. When a disagreement
arises, the difference often points to previously unrecognised assumptions or to some
interesting rationale for a different interpretation of the evidence. The use of ACH may
not result in the elimination of all the differences of opinion, but it can be a big step
Page 55 of 113
toward understanding these differences and determining which might be reconcilable
through further intelligence collection or research.
Mind Mapping – This process can graphically display the logical relationship between
each element of an argument (see page 90) and, if two sides agree to work together to
create a single Mind Map, with the rationale both for and against a given conclusion.
Such a map would show where the two sides agree, and where they diverge, and why.
The visual representation of the argument makes it easier to recognise weaknesses in
opposing arguments. An alternative approach might be to focus the discussion on
alternative contrasting Mind Maps.
Mutual Understanding – When analysts disagree, the disagreement is often exacerbated
by the fact that they have a limited understanding of the other side’s position and logical
reasoning. The Mutual Understanding approach addresses this problem directly. After
an exchange of information on their positions, the two sides meet together with a
facilitator, moderator, or decision maker. Side 1 is required to explain to Side 2 its
understanding of Side 2’s position. Side 1 must do this in a manner that satisfies Side 2
that its position is appropriately represented. The roles are then reversed, and Side 2
explains its understanding of Side 1’s position. This mutual exchange is often difficult to
do without really listening to and understanding the opposing view and what it is based
upon. Once all the analysts accurately understand each side’s position, they can discuss
their differences more rationally and with less emotion.
The Nosenko Approach – This process is named for a Soviet intelligence officer who
defected to the US in 1964, and the ground rules used by the CIA’s analytic team to try to
determine the “truth” about him. After reviewing the evidence, each officer on the team
identified those items of evidence thought to be of critical importance in making a
judgement on Nosenko’s bona fides. Any item that one officer stipulated as critically
important had to be addressed by each of the other members. These ground rules can be
applied in any effort to abate a long-standing analytic controversy. The key point that
makes the rules work is the requirement that each side must directly address the issues
that are important to the other side and thereby come to understand the other’s
perspective. This process guards against the common propensity of analysts to make
their own arguments and then simply dismiss the other side’s arguments as unworthy of
consideration.
Encouraging Consensus.43
One way to achieve a consensus is through the realization of a series of little consensuses. Using
this iterative approach, the following questions should be asked and answered - in turn - by the
group:
What is the problem that needs to be addressed?
What caused the problem?
What are the alternative solutions?
What are the criteria for a good solution?
What is the best solution?
43
Daniel Lapensée, CHRA, YouTube – Comment render les réunions efficaces, 3 Jul 2011
Page 56 of 113
Use Structured Analytic Techniques
Why use Structured Techniques?
The analytic tools and techniques that have been mentioned so far in this Aide Memoire and
which follow all, potentially, have three positive attributes:
First, they may instil rigor in the process of analysis by providing a checklist that can be
followed to guide the process.
Second, they may actually bring into existence a better analysis than would have been
produced without them.
Finally, they may enhance the persuasiveness of arguments and provide a useful
visualisation when presenting the assessment.
Be careful though, using the wrong technique, using faulty or incomplete data, or allowing an
inappropriate assumption or mindset to prevail will almost certainly result in poorer analysis and
wasted time.
Categories of Techniques
There are a number of ways of dividing structured analytic techniques into groups. One of these
is shown on the back of the IALP’s Best Practices for Intelligence Analysts card. In that schema,
the techniques are classified as either:
Diagnostic – these techniques are primarily aimed at making analytic arguments,
assumptions, or intelligence gaps more transparent.
Challenge, or contrarian – these techniques are intended to explicitly challenge current
thinking.
Imaginative – these are thinking techniques that aim to develop new insights, different
perspectives and/or alternative outcomes.
The structured analytic techniques in this Aide Memoire, however, have been classified
according to how they contribute to the analytic tradecraft process. To this end, nine separate
groupings have been identified as follows (see also the CMap on page 57):
Use Work Groups – these techniques are various ways of organising analytic teams in
order to tackle intelligence problems.
Develop – these techniques are intended to help analysts explore issues and come up with
factors and alternatives.
Evaluate – these techniques help analysts evaluate the usefulness (veracity, or
appropriateness) of evidence, assumptions and indicators.
Explore Networks – these two techniques use link processes to explore networks and
relationships.
Compare – these techniques rank things according to the factors or criteria.
Generate Scenarios – these techniques use factors to develop scenarios.
Understand Possibilities – these techniques use factors to explore possibilities.
Demonstrate – these techniques will help an analyst illustrate or sequence analysis in a
compelling or understandable way.
Page 57 of 113
Review Pre-Publication – these techniques provide ways to examine analysis before it is
published.
The techniques are, below, shown alphabetically and by family group:
Technique Page
Alternative Futures Analysis. 83
Analysis of Competing Hypotheses 74
Bow-Tie Technique 86
Chronologies and Timelines 89
Cone of Plausibility 81
Counterfactual Reasoning 85
Cross-Impact Matrix 77
Change Analysis 80
Data Diagnostic 69
Decision Trees 92
Delphi Method 63
Devil’s Advocacy 61
Dialectic Inquiry 94
Environmental Scanning 64
Force Field Analysis 84
High Impact – Low
Probability 93
Hypothesis Generator 66
Hypothesis Review Technique 67
Indicators of Change 72
Indications Validator 73
Key Assumptions Check 70
Link Charts 76
Matrices 91
Mind Mapping 90
Outside-In Thinking 65
Pre-Mortem Assessment 95
Quadrant Hypothesis Generation
Technique 82
Red Cell 60
Red Team 59
Reframing the Question 68
Social Network Analysis 75
Structured Brainstorming 62
Structured Comparison 78
Team A – Team B 58
Weighted Rankings 79
What If? Analysis 88
Structured Analytic Techniques Map
Page 58 of 113
Use Work Groups
Team A / Team B.
Purpose: To provide a self-check that confirms, refutes, or offers an alternative (competing) assessment.
Description: Team A/Team B is a competition, akin to a debate. However, unlike Devil’s Advocacy,
which simply tries to refute an argument, it permits the expression of different views. Separate teams are
assigned the same problem. They may be told to take opposite sides or they may be left to arrive at
independent conclusions.
Process: When debate is polarized between two views, the Team A/Team B approach helps clarify the
differences between the two viewpoints. When it is suspected that analysts have developed an inflexible
mindset, it can force a re-evaluation.
The process has two phases, analysis and debate.
In the Analysis Phase:
Begin by identifying the two (or
more) competing hypotheses or points
of view.
Form teams (or designate individuals) to
develop the best case that can be made for
each hypothesis.
In the Debate Phase:
Both teams present their arguments
before an independent jury.
The teams critique each other’s
assessments then respond to criticisms.
The jury or decision maker listens, questions, and decides which team has the stronger case and may also
recommend actions to be taken, e.g. further research.
Strengths: Assigning analysts to the team that fits their existing view expedites the process and makes
for strong arguments. The reverse takes more time, but forcing analysts to argue “the other side” can often
make them more aware of their own assumptions, mindsets and biases.
The process can help opposing experts see the merit in each other’s arguments.
It also allows those with opposing views the opportunity to express their views and feel that they have
been heard.
For the decision maker, this technique exposes and explains important analytic differences within the
expert community. Senior officials often learn more by weighing well-argued conflicting views than
from reading an assessment that masks substantive differences or drives analysts to the lowest common
denominator.
Weaknesses: The process requires time and resources. Competition may lead to workplace acrimony and
anchoring. Results may be skewed if one team has a political agenda.
Page 59 of 113
Use Work Groups
Red Team.
Purpose: To produce assessments which represent probable viewpoints and intentions of the subjects of
the analysis.
Description: Red Teams have been used historically in planning and implementing military exercises, in
which the Red Team plays the opposing force. A Red Team is a group brought together to simulate the
intentions, planning, and sometimes the capabilities of the subject group. A Red Team can produce
predictions of possible or probable future actions by the subjects. A Red Team can also identify
vulnerabilities in our security, planning, and other aspects of our capabilities which the subject could
exploit.
Analytic Red Teams in intelligence are useful in predictive assessments and alternative analyses,
potentially at both the tactical and strategic levels. This is not the only possible use of this technique, for
example operational Red Teams which employ hacking techniques are used to actively test IT security.
Members of a Red Team need to have a rich understanding of the subjective and objective worlds of the
subjects, including: cultural norms and values, motives, operating doctrine, operational environment,
sensitivity to risk, and other relevant factors such as their subjective understanding of our intentions and
capabilities. Operational Red Teams require intelligence support to successfully imitate their subjects’
intentions and capabilities.
Strengths: A well-informed Red Team can give valuable insights into the courses of action available and
acceptable to an adversary. This can provide a balance against biases, especially mirror imaging.
Weaknesses: Without a rich understanding of the subjects, members of a Red Team may tend to fall into
mirror imaging and so produce confirmation of potentially false understandings. There can also be
significant organizational challenges to using a Red Team approach if the purpose is not widely
understood or if individuals are too invested in the positions being tested.
Page 60 of 113
Use Work Groups
Red Cell.
Purpose: A Red Cell is an internal think tank created to help an organization develop broader peripheral
vision and avoid surprise.
Description: In response to 911, the CIA created a Red Cell that would think unconventionally about the
full range of relevant analytic issues. Their short “think pieces” about plausible events for which there
was currently little or no evidence were intended to provoke thought rather than to provide authoritative
assessment.
A Red Cell does not automatically take the opposite or contrarian view – it is not a challenge technique. It
merely tries to raise awareness.
The process starts with a question such as: “Could the UN in New York be attacked using garbage barges
laden with explosives?”
Any feasible question is OK – although, ideally it should be one that no one else has asked. The next step
is generating a series of events or scenario that would lead to the event occurring, i.e. “What if?” analysis.
Finally, a short but (hopefully) convincing argument is presented that summarizes the evidence and
reasoning.
A Red Cell can be set-up permanently or on an ad hoc basis to deal with specific issues (in which case it
might be referred to as a “Team A / Team B” exercise or a “Red Team” if it simulates the actions or
decision-making process of an adversary).
If successful, Red Cell reports persuade experts, analysts, and decision makers to re-examine their views
and re-task resources to prove or disprove the scenario advanced by the Red Cell.
A Red Cell needs a diverse group of people who are “out of the box” thinkers. Not afraid to speak out or
appear the fool, they should be ingenious and resourceful. Members could include psychologists,
philosophers, professors, and analysts from different units and agencies who think differently from the
ones who typically work on the account (i.e. file / issue / problem). Outside technical experts may be
included or consulted (depending on the problem).
To ensure a continual flow of fresh insights, new members can be rotated in every three months.
Management should also give permanent Red Cell units lots of latitude (two thirds of CIA Red Cell
reports are self-initiated).
Strengths: Helps to combat assumptions, mindsets, and biases.
Weaknesses: A dedicated Red Cell that constantly critiques or “shows-up” the analysis of others, can
create internal tensions. One that “colours too far outside the lines” risks being ignored or mocked. Red
Cell reports may provoke hostility, especially within conservative, hierarchical organizations. “As if we
didn’t have enough problems already,” could be one reaction. If Red Cell reports undermining
mainstream analysis are distributed externally, the agency could appear indecisive. Small organizations
may not be able to afford removing staff from their regular duties to work in a Red Cell. Some argue
creativity cannot be structured.
Page 61 of 113
Use Work Groups
Devil’s Advocacy.
Purpose: An analytical technique used to test an argument by attempting to prove it flawed or
fundamentally incorrect.
Description: The original advocatus diaboli [Devil's Advocate] was, from 1587 to 1983, part of the
Roman Catholic Church's formal process of recognizing individuals as saints. As the name suggests, his
role was to take the opposing view against his opponent the advocatus Dei [God's advocate] promoting an
individual as a saint.
This competitive approach to testing a hypothesis or argument is also seen in the structured confrontations
between prosecution and defence in courts of law.
It can also be placed in the context of Hegelian dialectic, in which thesis generates antithesis, leading to a
higher synthesis.
A Devil's Advocate exercise can be effective in testing hypotheses in terms of finding any weaknesses in
logic.
This technique can be used at several stages during the development of an intelligence assessment. For
example, a Devil's Advocate could review the project plan, the working hypotheses, and/or the final draft
of the assessment.
While the typical approach to multiple competing hypotheses is based on analysts developing and testing
their own hypotheses, the Devil's Advocate technique brings in an outsider to test the hypotheses. The
individual or team playing the role of Devil's Advocate develops and presents arguments against the
hypotheses and conclusions presented in the assessment under review.
An individual or team playing the Devil's Advocate will need to study not only the hypotheses or
assessment it is testing, but must also study the supporting information.
Strengths: An effective Devil's Advocate can find flaws in a hypothesis or assessment which might
otherwise not be noticed by the authors, and a hypothesis which has survived a thorough and rigorous
testing through this process can be given more credibility.
A Devil’s Advocate also reduces “group think.”
Weaknesses: A Devil's Advocate can suffer from the same biases as the author of the hypothesis, and so
miss the same flaws.
Use of the Devil's Advocate approach can generate conflict and morale problems in an intelligence group
if the individuals involved identify too closely with the positions being argued and tested.
A weak Devil's Advocate may also create a misleading impression that all sides of an issue have been
considered, when in reality they have not. This could lead to false confidence and over-commitment.
Page 62 of 113
Develop
Structured Brainstorming.44 Purpose: A form of brainstorming that generates new analytic ideas, hypotheses, or concepts through an
unconstrained individual or group process.
Description: This technique works best when an individual is willing to work as part of a group to
develop multiple ideas, hypotheses, or concepts. It can be used either at the beginning of an analytic
project to help generate the initial hypotheses or at a later stage if the initial result proves inadequate. New
information may be found that could cause the analyst to return to this technique to integrate it into the
existing hypotheses.
Process: Creative thinking works best when a trained facilitator is available to ensure the session is
fruitful. The creative thinking process actually consists of two phases: a divergent phase, where group
members create new ideas via brainstorming, and a convergent phase, where group members cluster ideas
for review, consolidation, and follow-up action.
Divergent Phase
Organize the group. Group members should come from a variety of backgrounds (cross
fertilization is important). Cognitive diversity, different points of view, and a wide range of
experience are important. Small groups tend to function better than large ones; five to seven
participants is a good target.
Focus on a specific topic or question. It should not be so broad that no solution is possible or so
narrow that creativity won’t help. Make clear to all members in advance that discussion will not
be constrained by current positions or available evidence.
Have everyone write down at least one idea before discussion starts. Use paper, white boards,
or Post-it notes to record ideas. That will allow easy clustering of ideas during the convergent
phase.
Have the group verbally generate as many ideas as possible. When a group has one or more
strong personalities, the facilitator can have the members stop all verbalization and write their
ideas down and post them where others can read them and build on any idea. Listen closely as
others talk; this will help generate ideas. Suspend judgement; do not eliminate ideas; what looks
crazy at first may become valuable later, after more thought or when new data is received.
Let the first session last for 45-60 minutes or until a noticeable decline in activity takes
place. Then take a break. Keep going for two more sessions, ending each when the activity falls
off. After the third such period, it is time to stop the divergent phase.
Convergent Phase
Group the ideas by theme, then set aside any that do not easily fit with any group. Then
through voting or other means, select the themes or outliers that deserve further attention.
After the session is over, have the individuals spend time alone to silently review the
submission and consider: Which of the alternatives are reasonable and would meet the goals?
What are the alternative’s shortcomings? What are the alternative’s benefits?
44
Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006,
pp 21-24 and Handbook of Analytic Tools & Techniques, Pherson, 2008, pp19-20
Page 63 of 113
Develop
Delphi Method.
Purpose: To produce reliable consensus judgements from a large and diverse set of experts.
Description: The Delphi method is an iterative survey approach to generating consensus judgements.
The progressive series of questionnaires is a structured, anonymous debate among the participants.
The first set of questionnaires is sent out to the panel of experts who provide their individual judgements.
The next set of questionnaires includes anonymous summaries of the responses to the previous
questionnaire, the panel members are asked to consider these other responses and to revise or maintain
their own judgements accordingly. A reliable consensus of opinion usually develops in the second
iteration, but the process can be repeated further if necessary. Sometimes the survey will reveal stable
disagreement in the form of two or more factions holding incompatible opinions.
The Delphi method can be used in predictive assessments, in priority setting assessments, and in
generating expertise-based analytical tools.
A successful Delphi depends on good survey design, sound analysis of the survey results, and on
involving respondents with valid and relevant expertise.
Strengths: Studies of Delphi method have shown it to produce more reliable consensus judgements than
face-to-face meetings. A Delphi survey can also be more efficient and economical that arranging and
facilitating a meeting / workshop of the appropriate experts.
Weaknesses: Errors in survey design, survey results analysis, or in selecting the participants can lead to
invalid results.
Page 64 of 113
Develop
Environmental Scanning. Purpose: Environmental Scanning provide checklists that may help an analyst explore a subject area in
broad terms.
Process: Having selected an appropriate environmental scanning process, use each of its component parts
to explore the matter under exploration.
SWOT
Strengths
Weaknesses
Opportunities
Threats
Activity Systems Model
45 See the diagram on the right.
STEMPLES46
47
Social (culture, attitudes/perceptions, education, population, health, welfare, etc.)
Technological (developments, funding, access to technology, patents/licensing, IT, etc.)
Environmental (climate, living conditions, impact of weather, etc.)
Military (capabilities, developments, doctrine, control & command, etc.)
Political (leadership, political system, policy, pressure groups, elections, etc.)
Legal (current & future legislation, regulatory processes, judicial system, etc.)
Economic (internal economy, inflation, global markets, industry, interest rates, etc.)
Security (police, reforms, private companies, terrorism, criminal networks, etc.)
Understanding Groups48
What defines this group (geography, history, purpose, etc.)?
Are there internal factions (religious, ethnic, linguistic, generational, etc.)?
How is power distributed (matriarchic, patriarchal, wealth, fear, moral authority, etc.)?
What motivates and inhibits individual conduct?
What pressures and stresses are present in the group?
How is the culture changing?
What unusual values, attitudes or beliefs are present? What are the taboos?
How are outsiders viewed?
What are the capabilities and intentions of this group?
Do members of this group think as you do?
45
Adapted from the US DIA’s “Activity Systems Model (expanded)” demonstrated by JMITC instructors at the Feb
2013 5-Eyes Analytic Training Workshop (they had derived it from Yjro Engestrom). 46
Copied from the DIS Handbook Quick Wins for Busy Analysts, 2012, p 9 47
One could also consider psychological, ethical, demographic and other factors. 48
Based on the check list developed by John Pyrik for use in the Canadian Interdepartmental Intelligence Analyst
Learning Program.
Tools
Actor (s) Task (s)
Roles RulesOrganisation
Community Governance
Context
Motivation Outcomes
Tools
Actor (s) Task (s)Actor (s) Task (s)
Roles RulesOrganisationRoles RulesOrganisation
Community GovernanceCommunity Governance
Context
MotivationMotivation OutcomesOutcomes
Page 65 of 113
Develop
Outside-In Thinking.49 Purpose: Outside-In Thinking is used to identify the range of systemic forces, factors, and trends that
would have an impact on shaping an issue, allowing analysts to incorporate this broader conceptual
framework into their analysis.
Process: Once a generic description of the problem or phenomenon under study has been generated, the
analyst can use an environmental scanning process (see page 64) to trigger new ideas. The process is:
First, to list all of the factors over which the
subject may be able to exert limited influence
(such as resources, partners, methodologies,
etc.).
Then, to list all of the forces and factors that
could have an impact on the topic, but over
which the subject can exert little or no influence
(such as historical precedent, the growth of the
Internet, etc.).
Then the analyst can assess, specifically, how each of these forces and factors might have an impact on
the problem and generate new collection taskings to fill the information gaps.
The technique is most useful in the early stages of the analytic process when analysts need to identify all
the critical factors that could influence how a particular situation will develop as well as potential gaps in
the reporting. The technique is also useful if a large database is being assembled by helping to ensure that
an important field is not forgotten in the database architecture.
Strengths: Outside-In Thinking encourages analysts to move “outside their inbox” to rethink a problem
by employing a broader conceptual framework. By casting one’s net broadly, analysts are more likely to
see an important dynamic or to include a relevant alternative hypothesis. The process can provide new
insights and uncover relationships that were not evident from the reporting. In doing so, the technique
helps analysts think in terms that extend beyond day-to-day reporting to identify previously un-
considered, but fundamental forces and factors.
49
Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.33
What can be influenced?
What cannot be influenced?
Page 66 of 113
Develop
Hypothesis Generator.50
Purpose: The Hypothesis Generator provides a structured mechanism for generating a wide array of
hypotheses.
Description: This process uses a critical examination of a lead hypothesis to generate plausible
alternatives.
Process: First, crisply define the issue, activity, or behaviour that is subject to examination and establish a
lead hypothesis. (The lead hypothesis could be the one you were given, the most obvious explanation, or
the conventional wisdom.) Then:
Critically examine the lead hypothesis by identifying and listing its key elements. Use Who,
What, When, Where, Why, How and So What (in this or any other order) to evaluate all
dimensions of the hypothesis (note that some of these questions may not be appropriate for the
task at hand).
Generate plausible alternatives for each key element (see the example).
Generate a list of all possible permutations (see the box below).
Discard any permutation that simply makes no sense.
Evaluate the credibility of the
remaining permutations by
challenging the key assumptions of
each component.
Assign a “credibility score” to each
permutation using a 1 to 5 point scale
(see box on right).
Sort the permutations, listing them
from most to least credible (not
shown).
The permutations at the top of the list become the alternative hypotheses most deserving of
attention.
50
Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.12
Etc.EtcEtcEtc
5Permutation 1-3-33.35Permutation 1-3-23.22.35Permutation 1-3-13.1
4Permutation 1-2-33.32Permutation 1-2-23.22.21.11Permutation 1-2-13.1
4Permutation 1-1-33.32Permutation 1-1-23.22.12Permutation 1-1-13.1
Etc.EtcEtcEtc
5Permutation 1-3-33.35Permutation 1-3-23.22.35Permutation 1-3-13.1
4Permutation 1-2-33.32Permutation 1-2-23.22.21.11Permutation 1-2-13.1
4Permutation 1-1-33.32Permutation 1-1-23.22.12Permutation 1-1-13.1
Example:
Terrorists are trying to harm us by providing illegal drugs through the Internet.
Element 1
1. terrorists,
2. drug companies,
3. entrepreneurs,
4. etc.
Element 2
1. harm us,
2. make money,
3. sabotage health care
programs,
4. etc.
Element 3
1. illegal drugs,
2. illegal prescription drugs,
3. legal prescription drugs,
4. etc.
Example:
Terrorists are trying to harm us by providing illegal drugs through the Internet.
Element 1
1.1 terrorists,
1.2 drug companies,
1.3 entrepreneurs,
1.4 etc.
Element 2
2.1 harm us,
2.2 make money,
2.3 sabotage health care
programs,
2.4 etc.
Element 3
3.1 illegal drugs,
3.2 illegal prescription drugs,
3.3 legal prescription drugs,
3.4. etc.
Example:
Terrorists are trying to harm us by providing illegal drugs through the Internet.
Element 1
1. terrorists,
2. drug companies,
3. entrepreneurs,
4. etc.
Element 2
1. harm us,
2. make money,
3. sabotage health care
programs,
4. etc.
Element 3
1. illegal drugs,
2. illegal prescription drugs,
3. legal prescription drugs,
4. etc.
Example:
Terrorists are trying to harm us by providing illegal drugs through the Internet.
Element 1
1.1 terrorists,
1.2 drug companies,
1.3 entrepreneurs,
1.4 etc.
Element 2
2.1 harm us,
2.2 make money,
2.3 sabotage health care
programs,
2.4 etc.
Element 3
3.1 illegal drugs,
3.2 illegal prescription drugs,
3.3 legal prescription drugs,
3.4. etc.
Page 67 of 113
Develop
Hypothesis Review Technique.51 Purpose: An effective technique to mitigate mirror-imaging bias and to understand an adversary’s
potential mitigation strategies.
Description: After generating hypotheses to answer a given question, use this technique to gain a better
understanding of alternatives from the adversary’s point of view.
Process: For each alternative or
hypothesis to be reviewed, perform the
following steps from the point of view of
the adversary. Ensure all the steps for one
alternative are complete before going on
to the next.
List all the benefits or pluses for
the alternative being reviewed
from the adversary’s point of
view. (Why would this alternative
be a good choice?)
List all the risks or minuses for
the alternative being reviewed
from the adversary’s point of
view. (What detracts from this
alternative being a good choice?)
Review and consolidate the risks,
merging and eliminating as
appropriate.
Identify possible risk mitigation strategies that the adversary may adopt for each risk or minus.
Compare the benefits and unalterable risks for all options.
Finally, consider the results of this review on the overall analysis – amend if necessary.
Strengths: This technique can mitigate mirror-imaging bias and the natural negative bias. It can be
applied to any problem at any point in the analytic process as a simple, fast, and effective technique to
gain insight into the cultural-based adversary’s point of view, and of the potential implications of the
various options.
Weaknesses: This method can generate false impressions of a given alternative or the benefit-versus-risk
calculus if the analyst does not have an adequate understanding of the adversary’s culture-based point of
view.
Note: A similar, but slightly more complicated process – called the Adversarial Options Matrix – is
shown at the bottom of page 91.
51
Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2008,
pp 27-29
Plusses Minuses Mitigation Strategies
First Hypothesis or Alternative
1.
2.
3.
Etc.
1.
2.
3.
Etc.
1.a
1.b
2.a
2.b
3.a
3.b
Etc.
Second Hypothesis or Alternative
Etc. Etc. Etc.
Third Hypothesis or Alternative
Etc. Etc. Etc.
Page 68 of 113
Develop
Reframing the Question.52 Purpose. This reframing technique provides a vehicle through which analysts can experiment with
different ways of defining an issue.
Description. Reframing the Question should be used whenever an analyst begins to assess a new issue or
problem, or begins a new research endeavour. Proper issue identification can save a great deal of time
and effort that is easily misspent on research and analysis of a poorly stated issue.
Process. Start with the best possible statement of the issue and then re-consider that statement as follows:
Paraphrase. Redefine the issue without losing the original meaning. Review the results to see if
they provide a better foundation upon which to conduct the research and assessment to gain the
best answer. Example: the original question, How much of a role does Aung Sung Sui Kyi play in
the ongoing unrest in Burma? is rephrased as, How active is the NLD headed by Aung Sung Sui
Kyi in the current antigovernment riots in Burma?
180 Degrees. Turn the issue on its head. Is the issue the one asked or the opposite of it? Example:
the original question, How much of the PLA ground capability would be involved in an initial
assault on Taiwan? is rephrased as, How much of the PLA ground capability would NOT be
involved in the initial Taiwan assault?
Broaden the Focus. Instead of focusing on only one piece of the puzzle, step back and look at
several pieces together. What is the issue before you connected to? Example: the original
question, How corrupt is President Musharraf? leads to the question, How corrupt is the
Pakistani government?
Narrow the Focus. Can the issue be broken down further? Take the question and ask about the
components that make up the problem. Example: the original question, Will the EU ratify a new
constitution? can be broken down to, How do individual member states view the new EU
constitution?
Redirect the Focus. What outside forces impinge on this issue? Is deception involved? Example:
the original question, What are the terrorist threats against the US homeland? is revised to, What
opportunities are there to interdict terrorist plans?
Ask Why. Ask why of the initial issue or question. Develop a new question based on the answer.
Then ask why of the second question and develop a new question based on that answer. Repeat
this process until you believe the real problem emerges. This process is especially effective in
generating possible alternative answers.
52
This section is based on the DIA Analytic Methodologies, A Tradecraft Primer: Basic Structured Analytic
Techniques, First Edition, March 2008, pp.5-6
Page 69 of 113
Evaluate
Data Diagnostic.
Purpose. The purpose of the Data Diagnostic process is to give analysts a process to use when
considering the veracity and usefulness of data.
Description. A thorough diagnosis of the available data (evidence, information, opinions, etc.) can often
reveal shortcomings that, once recognised, can be mitigated. To this end, the analyst should consider the
provider of the information, the information itself, the relevance of the information to the problem at
hand, and the probability of the existence of denial and/or deception.
Process. The Data Diagnostic process is divided into four parts:
Provider Diagnostic. Evaluation of the provider should consider:
o The agency, or witness, or source of the information
o And the handler (editor, agent handler, translator, processor, interpreter, etc.)
o In terms of:
Access and
Motivation.
Information Diagnostic. The information (data, evidence, opinions, etc.) itself should be
examined by considering its:
o Quality
Source access
Corroborability
Consistency
Completeness
o Logic
Main purpose
Key question
Important info
Inferences
Concepts
Assumptions
Implications (positive and negative)
Point of View
Relevance. A simplistic approach to relevance is to examine the salience of the information
(data, evidence, opinions, etc.) to the matter at hand (i.e. is it related, consistent, complete?).
Beyond salience, however, relevance can be examined through the lenses distorted by the
influence of biases that often prompt analysts to give credence to material inappropriately. Some
of these cognitive pitfalls are: the vividness criterion, the oversensitivity to consistency, the bias
favouring centralised direction, the similarity of cause and effect, internal vs. external causes of
behaviour, and the illusory correlation.
Denial and Deception. Finally, the possibility of denial and deception should be examined
through five sets of criteria:
o Does the potential deceiver have Motive, Opportunity, and Means (MOM) to deceive?
o Would this deception be consistent with Past Opposition Practices (POP)?
o Do we have cause for concern regarding the Manipulability of Sources (MOSES)?
o What can we learn from our Evaluation of Evidence (EVE)?
o Is the Source of the information simply Naïve (SON)?
Page 70 of 113
Evaluate
Key Assumptions Check.53
Purpose: List and review the key working assumptions on which fundamental judgements rest.
Description: A Key Assumptions Check is most
useful at the beginning of an analytic project when
an hour or two can be instrumental in ensuring that
the impending assessment does not rest on flawed
premises. Rechecking assumptions can also be
valuable at any time prior to finalising judgements
to insure that the assessment does not rest on flawed
premises.
Process: Checking for key assumptions requires
analysts to consider how their analysis depends on
the validity of certain premises, which they do not
routinely question or believe to be in doubt. A four
step process can help with this challenge:
Review what the current analytic line on the
issue appears to be; write it down.
Articulate all the premises, both stated and
unstated in finished intelligence, which are
accepted as true for this analytic line to be
valid.
Challenge each assumption, probing why it “must” be true and whether it remains valid under all
conditions. The questions on the right will help in this regard.
Refine the list of key assumptions to contain only those that “must be true” to sustain the analytic
line; consider under what conditions or in the face of what information these assumptions might
not hold.
Alternatively - How to do it54
1. Identify your analytic line to be tested. It might look as follows:
RED is GREEN’s most important military supplier. RED has also been pivotal in assisting
GREEN with its WMD programmes which are now reaching maturity.
2. List all of the key assumptions that you believe underpin the analytic line, i.e. those that are accepted as
being true for the conclusions to be valid. For the example above these would look as follows:
Number Assumptions
1 RED is supplying GREEN militarily
2 GREEN has no other significant military supplier
3 RED has provided GREEN with non-military goods and training
53
Developed from the CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence
Analysis, Jun 2005, pp 7-9 54
Copied from the UK Defence Intelligence Quick Wins for Busy Analysts, 2012, pp 18-19
Questions to ask during this process:
o How much confidence exists that
this assumption is correct?
o What explains the degree of
confidence in this assumption?
o Is this key assumption more likely a
key uncertainty or key factor?
o Could this assumption have been
true in the past, but not now?
o If this assumption proves to be
wrong, would the analytic line be
significantly impacted?
o Has this process identified new
variables that need further analysis?
Page 71 of 113
4 GREEN has WMD programmes
5 GREEN depends on RED assistance for its WMD programmes
6 GREEN’s WMD programmes are reaching maturity
7 ...etc.
After you have developed as complete a list as you can, go back and critically examine each assumption
using the following questions to aid your thinking:
• If it were false, how seriously would this undermine the analytic line?
• How much confidence do you have that this assumption is valid?
- Why do you have this degree of confidence?
- Under what circumstances might this assumption be false?
- Could it have been true in the past but no longer true today?
- What would we expect to see if this assumption were true?
- Why aren’t we seeing these indicators?
Based on these, score each assumption according to two criteria:
• RELEVANCE:
- Largely irrelevant to analytic line (0)
- Important - analytic line would be significantly less likely if assumption were false (1)
- Essential - analytic line cannot be true without assumption (2)
• SUPPORT:
- Unsupported or very questionable (0)
- Correct with some caveats (1)
- Solid (2)
3. We are looking for the shaky, load-bearing assumptions. Find the assumptions which score highest for
“relevance”. Of these, the assumptions with the lowest “support” scores are the key uncertainties. A
matrix template like the one below should be used to filter your key assumptions and provide a clear
structure for the exercise. Use a comments column to record the rationale behind the results of your
confidence check. This might relate to the quality and quantity of evidence, reliability of sources etc.
In this example, the scores suggest that items 4 and 5 require revisiting as they are essential to the analytic
line but unsupported. Item 3 can be ignored.
Consider whether the key uncertainties identified have revealed collection requirements. The number of
key uncertainties will also dictate whether/how much the analytic line requires further research and
analysis, including contact with collectors, to ensure it is as robust as possible and accurately reflects the
available information.
Number Assumption Relevance Support
1 RED is supplying GREEN militarily 2 2
2 GREEN has no other significant military supplier 1 1
3 RED has provided GREEN with non-military goods and
training 0 2
4 GREEN has WMD programmes 2 1
5 GREEN depends on RED assistance for its WMD
programmes 2 0
Page 72 of 113
Evaluate
Indicators of Change.55 Purpose: To periodically review a list of observable events or trends to track events, monitor targets, spot
emerging trends, and warn of anticipated change.
Description: An analyst or team creates a list of indicators or
signposts of observable events that could be expected to
become apparent if a postulated situation is developing. The
technique can be used whenever an analyst needs to track an
event over time or monitor and evaluate changes.
Process: Whether used alone, or in combination with other
structured techniques, the process is the same:
Identify a set of competing hypotheses or scenarios.
Create separate lists of potential activities, statements,
or events that are expected to become manifest for
each hypothesis or scenario – and preferably only for
that one scenario or hypothesis.
Regularly review and update the indicators lists to see
which are changing.
Identify the most likely or most correct hypotheses or
scenarios, based on the number of changed indicators
that are observed.
Developing sets of indicators for each scenario or hypothesis
may be useful in distinguishing whether or not a development is
emerging or not emerging.
Strengths: By providing an objective baseline for tracking
events or targets, indicators instil rigor into the analytic process
and enhance the credibility of analytic judgements. Including an
indicators list in a finished product builds a more concrete case
for the analytical judgements and may help a client track
developments. By laying out a list of critical variables, analysts
will also be generating hypotheses regarding why they expect to
see the presence of such factors. In so doing, analysts make the analytic line much more transparent and
available for scrutiny by others.
Weaknesses: Analysts can become fixated on watching for the identified indicators, missing signals or
developments that are not anticipated.
55
Developed from the CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence
Analysis, Jun 2005, pp 12-13
Start Point
Indicator
Possible Path
Scenario
Start Point
Indicator
Possible Path
Scenario
Page 73 of 113
Evaluate
Indicators Validator.56
Purpose. This is a simple tool for assessing the diagnostic power of
indicators.
Description. Once the analyst (or the analytic team) has created a set of
alternative scenarios (or future worlds), and has also generated a set of
indicators for each scenario (or world), an indications validation process
may be undertaken to help determine the diagnostic value of the
indicators.
Process. In this process, the analysts populate a matrix similar to that
used for the Analysis of Competing Hypotheses (see page 74), involving the
following steps:
List the scenarios or worlds on the top of the matrix (as is done for
hypotheses in ACH).
List the indicators on the left side of the matrix (as is done with evidence in
ACH).
In each cell of the matrix, assess whether, for each particular
scenario, the indicator is:
o Highly likely to appear,
o Likely to appear,
o Could appear,
o Unlikely to appear, or
o Highly unlikely to appear.
Determine the diagnostic value of each indicator by assigning
a numeric value to each cell. If the “home” indicators are rated
HL for each attendant scenario, then scenarios with an L have
a value of 1, C = 2, U = 3 and HU = 4. (If the “home”
indicator is only rated as L [D and F in the example], then each
of the other values are one point lower.) The indicators with
the highest total score are the most discriminating, while
indicators with low scores are of little use.
Re-sort the indicators placing the most discriminating
indicators on the top of the matrix and the least discriminating
indicators at the bottom where they can be eliminated.
With the non-discriminating indicators eliminated, the analyst
should review the re-ordered list of indicators. If most indicators
for a particular scenario have been eliminated, the analyst should
develop new, and more diagnostic, indicators for that scenario (and
recheck their diagnostic value by applying the Indicators Validator
process to them as well).
56
Based on Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for Intelligence
Analysts, copyright 2011 by CQ Press, and an Indicators Validator paper, Jan 2009, by Randy Pherson.
Scenario Indicator Value
1. A. 1
B. 1
2. C. 1
D. 2
3. E. 1
F. 2
4. G. 1
H. 1
Highly Likely
Likely
Could
Unlikely
Highly Unlikely
Highly Likely
Likely
Could
Unlikely
Highly Unlikely
Scenario
Indicator 1. 2. 3. 4.
A. (6) HL C C C
B. (9) HL C U HU
C. (11) U HL HU HU
D. (9) HU L HU HU
E. (12) HU HU HL HU
F. (9) HU HU L HU
G. (11) HU U HU HL
H. (10) C HU HU HL
Scenario
Indicator 1. 2. 3. 4.
A. HL
B. HL
C. HL
D. L
E. HL
F. L
G. HL
H. HL
Page 74 of 113
Evaluate
Analysis of Competing Hypotheses. (ACH). 57 58
Purpose: ACH is a tool which can aid judgement on important issues through the careful weighting of
alternative explanations or conclusions. It helps to overcome, or at least minimize, some of the cognitive
limitations that make prescient intelligence analysis so difficult to achieve. ACH should not be expected
to reveal the “right answer,” but it may help identify likely hypotheses and allow the elimination of less
likely ones.
Process: ACH is a process as follows:
Identify the issue to be examined and specify the possible hypotheses to be considered. These
should, in a perfect world, be both mutually exclusive and all encompassing.
Make a list of significant evidence and assumptions. That is, everything that is relevant to
evaluating the hypotheses – including evidence, arguments, assumptions, and the absence of
things one would expect to see if a hypotheses were true or false.
Determine the credibility score (high, medium or low) for each piece of evidence or assumption
by considering the source, the quality and logic of the information, its relevance, and the
potential for denial and deception.
Prepare a matrix, with the hypotheses across the top and the evidence/assumptions/etc. and its
corresponding credibility score down the side. Consider each piece of evidence or assumption
against each of the hypotheses and determine its diagnosticity. Is it consistent (C), inconsistent
(I), very inconsistent (II) or neutral (N)? Score the data. Inconsistent pieces receive
a score of two marks and very inconsistent
receive four (consistent and neutral pieces
receive no marks). For each inconsistent or
very inconsistent piece, the credibility
weighting is then considered, and one extra
mark is added if the weighting is medium
and two extra marks if it is high.
Total the scores at the bottom of each
column.
Draw tentative conclusions about the
relative likelihood of each hypothesis.
Analyse how sensitive your conclusion is to
a few critical items of evidence or
assumptions. Consider the consequences
for your analysis if that
evidence/assumption were wrong, or
misleading.
Report conclusions – remember, the process only identifies hypotheses which are least likely.
Software: This form of analysis can be done in MSWord in a table, using MSExcel or with the simple
and free ACH Tool, (http://www2.parc.com/istl/projects/ach/ach.html).
57
Developed from Psychology of Intelligence Analysis, Heuer, 2003, Chapter 8. 58
Scoring process developed from The Art of Intelligence, Lahneman and Arcos, 2014, p. 32.
Issue or Problem
Being Examined
Hypotheses
Ranking ( II / I / N / C )Diagnostic
Weighting
( H / M / L )
Evidence
Assumption
Argument
Absence 654321
Total
Hypothesis 6
Hypothesis 5
Hypothesis 4
Hypothesis 3
Hypothesis 2
Hypothesis 1
Issue or Problem
Being Examined
Hypotheses
Ranking ( II / I / N / C )Diagnostic
Weighting
( H / M / L )
Evidence
Assumption
Argument
Absence 654321
Total
Hypothesis 6
Hypothesis 5
Hypothesis 4
Hypothesis 3
Hypothesis 2
Hypothesis 1
Page 75 of 113
Explore Networks
Social Network Analysis.
Purpose: To reveal patterns of interaction and the allocation of social and personal capital within a group
of people.
Description: Social Network Analysis (SNA) uses sociograms and mathematical computations to reveal
significant aspects of behaviour in groups. The sociograms (network analysis charts) can show the
relative importance of individuals in a network, and the nature and relative strengths of the links between
individuals.
Concepts used in SNA include:
Centrality - the relative position of an
individual within a network. Centrality
reveals influence and control over
communications and other exchanges.
Density - the number of links in a network
as a percentage of the number of possible
links. In general, high density groups adapt
more easily to the loss of an individual,
while low density groups are harmed less
by an informant in their midst.
Redundancy - the extent to which two or more individuals are equivalent to each other. The
equivalence may be in terms of their network role, or their functional capabilities.
Betweenness - is a measure of connectiveness. Individuals or groups that have many
paths between them have higher betweenness than those that do not.
Cutpoints - individuals who are the sole connections between network components.
SNA is a useful technique in targeting intelligence. It can break groups into smaller, separate entities, and
can identify individuals whose removal would cause the greatest disruption in the group’s functioning.
Detailed information about the connections between individuals in the group under study is required,
including the nature and content of those connections.
Strengths: Identifies the structural strengths and vulnerabilities of a group. In groups with a hierarchical
structure, SNA reveals patterns of communication, exchange and influence. In groups without formal
hierarchy or structure, the social network is the primary organizing principle.
Weaknesses: Requires rich data about the content of links between individuals, which can be difficult to
acquire when studying covert groups.
Recommended Readings:
“Notions of Position in Social Network Analysis” pp 1-35 in Sociological Methodology, Vol 22.
Stephen P. Borgatti and Martin G. Everett, 1992.
“Destabilizing Networks” pp 79-92 in Connections 24(3). Kathleen M. Carley, Ju-Sung Lee, and
David Krackhardt. 2002.
4
10 5
2
13
81215
13
9
14
26256
716
21
2417181920
22 23
4
10 5
2
13
81215
13
9
14
26256
716
21
2417181920
22 23
Page 76 of 113
Explore Networks
Link Charts. Purpose: The objective of a link diagram is to graphically depict relationship data.
These diagrams aid investigators and analysts by uncovering, interpreting, and displaying complex
information in easily-understood chart form. (See also page 27.)
Process:
Assemble all of the information.
Identify entities of interest and their
associations.
Construct an association matrix and
populate it with the information of
interest.
Identify the nature of the associations.
Draw a preliminary network diagram.
Refine the diagram.
Finally, develop hypotheses.
While individual analysts, and projects, will
have different ways of showing data, the
following are recommended:
show organizations as boxes and people as circles (icons
may also be used),
place people within the box for their organizations,
link individuals and organizations with lines (solid for
confirmed links, dotted or dashed for suspected links),
use coloured links to represent phone calls, financial
transactions, commodity flows, etc.,
avoid crossing lines,
place key entities in the centre, and
include a legend or key.
Strengths: Link Charts:
integrate data from different sources,
highlight key relationships,
serve as a visual briefing aid for team members,
prosecutors and juries,
show when cases overlap (when multiple charts
are combined),
provide a cumulative snapshot of a case.
Software: The Analyst's Notebook (i2inc.com) – the
Cadillac of charting software (about US$4,400) – is used
by over 2,000 law enforcement, military and intelligence
organizations. http://www.visualanalytics.com/index.cfm.
Toolbar
PaletteChart
Area
Toolbar
PaletteChart
Area
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Certain
Association
Suspected
Known
Owner
Suspected
Owner
?
?
?
?
?
? Collection
Gap
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Association
Known
Owner
Suspected
Owner
Inferred
Relationship
Inferred
Relationship?
?
?
?
?
? Collection
Gap
?
?
?
?
?
? Collection
Gap
I
I
I
I
I
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Certain
Association
Suspected
Known
Owner
Suspected
Owner
?
?
?
?
?
? Collection
Gap
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Association
Known
Owner
Suspected
Owner
Inferred
Relationship
Inferred
Relationship?
?
?
?
?
? Collection
Gap
?
?
?
?
?
? Collection
Gap
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Certain
Association
Suspected
Known
Owner
Suspected
Owner
?
?
?
?
?
? Collection
Gap
?
?
?
?
?
? Collection
Gap
Perso
n 1
Perso
n 2
Perso
n 3
Perso
n 4
Act
ivity
1
Locat
ion
1
Locat
ion
2
Locat
ion
3
Locat
ion
4
Perso
n 5
Locat
ion
5
Veh
icle
1
Association
Known
Owner
Suspected
Owner
Inferred
Relationship
Inferred
Relationship?
?
?
?
?
? Collection
Gap
?
?
?
?
?
? Collection
Gap
I
I
I
I
I
Cabal
Bill’s
Diner
Mini Mall
Andy
Jim
George
Chuck
Bird
Business
Oma
Cabal
Bill’s
Diner
Mini Mall
AndyAndy
JimJim
George
ChuckChuck
Bird
Business
OmaOma
Cabal
Bill’s
Diner
Mini Mall
AndyAndy
JimJim
George
ChuckChuck
Bird
Business
OmaOma
Cabal
Bill’s
Diner
Mini Mall
AndyAndy
JimJim
George
ChuckChuck
Bird
Business
OmaOma
Page 77 of 113
Compare
Cross-Impact Matrix.59
Purpose: By comparing and contrasting the variables of a complex problem, analysts can systematically
examine how each factor in a particular context influences all other factors to which it appears to be
related.
Description: When analysts are estimating or forecasting future events,
they consider the dominant forces and potential future events that might
influence the outcome. The Cross-Impact Matrix provides a context for
discussing the relative influence of these forces or events and a structure
for managing that complexity.
Process: Having settled on a firm understanding of the issue at hand, the
Cross-Impact Matrix process is as follows:
Develop a list of variables and/or events that would likely have some effect on the issue being
studied.
Create a matrix and put the
variables or events down the
left side of the matrix and the
same variables or events
across the top.
Use the matrix to consider
and record the relationship
between each variable or
event and every other
variable or event. For
example:
o Does the presence of Variable 1 increase or diminish the influence of Variables 2, 3, 4,
etc.? Or, does the occurrence of Event 1 increase or decrease the likelihood of Events 2,
3, 4, etc.?
o If one variable does affect the other, record its magnitude in the variable’s row. (Note
that each relationship is considered twice. Variable 1’s influence on Variable 2 is
recorded in the Variable 1 row, while Variable 2’s influence on Variable 1 is recorded in
the Variable 2 row – and they might not be the same.)
Study the matrix to determine:
o The direction and the magnitude of the various relationships.
o Which variables have the greatest impact on others (in the example, variables 2 and 5
have the strongest influence).
o If there are any variables that might reinforce each other either directly (variables 2 and
4), or through a third variable (variables 3, 1 and 5). Combinations of variables that
reinforce each other can lead to surprisingly rapid changes in a situation.
Strengths: The Cross-Impact Matrix often reveals that variables or issues once believed to be simple and
independent are, in reality, interrelated.
59
Developed from Richards J. Heuer, Jr. And Randolph H. Pherson’s Structured Analytic Techniques for
Intelligence Analysts, pp 104-107, copyright 2011 by CQ Press.
Magnitude
++ Strong positive
+ Positive
Neutral
- Negative
-- Strong Negative
Variable 1 Variable 2 Variable 3 Variable 4 Variable 5
Variable 1 + ++
Variable 2 -- - ++ +
Variable 3 ++ - +
Variable 4 ++ +
Variable 5 -- + ++ +
Page 78 of 113
Compare
Structured Comparison. 60
Purpose: To rank-order priorities for complex issues with large sets of options.
Description: This technique uses sets of defined criteria to allow a formal comparison of entities or
phenomena. The set of attributes (criteria, qualities) and their values will be specific to each problem and
must be very carefully established.
Structured Comparison / Multiple Criteria
Analysis are approaches to decision making
used in the private sector, military and
government. In the context of intelligence
analysis, structured comparison tools have to be
specific to the decision-making problem. For
example, set of attributes designed to rank-order
organized crime groups in terms of their
strengths and weaknesses will not be valid for
terrorist groups.
The attribute set must be developed by the
appropriate range of subject matter experts in order to ensure relevance to the
problem. The criteria must have clear working definitions to calibrate the
technique, and so ensure the reliability of the results. Analysts will need to have a
good understanding of the criteria as defined in the technique in order to apply it.
Strengths: More objective, comprehensive, and reliable than unstructured
comparison exercises. The procedure can be repeated over time, and can be
applied by multiple groups of analysts; it increases credibility of rank ordering,
and permits auditing of a process.
Weaknesses: Time and resources are needed to create specific attribute sets. If the process and results
are not reviewed there is the potential for respondents to slant their input, whether consciously or
subconsciously, in an attempt to promote their preferred results.
Example: In order to develop the matrix above, high, medium, low and nil definitions of each of the
criteria (A, B, C, etc.) would need to be developed and agreed upon. Then, for each entity or
phenomenon an assessment would need to be made for each of the criteria to determine the appropriate
level. Colour coding and sorting could be done either manually, or using an IT solution.
60
Developed from Project Sleipnir – An Analytical Technique for Operational Priority Setting, a paper presented by
Steven J. Strang, RCMP, at the 2005 International Conference on Intelligence Analysis, McLean, VA,.May 2 to 4,
2005.
E
D
C
B
A
7651234Criteria
Qualities
Entities or Phenomenon
Complex Issue to be Examined
E
D
C
B
A
7651234Criteria
Qualities
Entities or Phenomenon
Complex Issue to be Examined
Unknown
Nil
Low
Medium
High
Unknown
Nil
Low
Medium
High
Page 79 of 113
Compare
Weighted Rankings.61
Purpose: A technique used by an individual or group to gain confidence in the assessment of available
alternatives by weighting criteria in importance.
Process: Having determined (using a separate
technique) the question to be answered, there are
eight steps to accomplish a weighted ranking review:
Identify all the major criteria for ranking,
Pair-rank the criteria by rating each
individual criteria against each other criteria
in turn, and
Select top several criteria and weight them
in percentiles.
Then, having identified the items to be ranked:
Construct a weighted-ranking matrix; enter
the items to be ranked, the
selected criteria and their
weights,
Pair-rank each of the items to
be ranked by each criterion
and record the votes,
Multiply the number of votes
by criterion’s weight,
Add the weighted values for
each item and total, and
Determine the final rankings.
Conclude by conducting a sanity
check of the results and reviewing the
impact of the weighted criteria on the
final result.
Example: To answer the question – What is the best method for conducting a terrorist attack? The
top box identifies, pair ranks, and weights the major criteria (what does best mean in this context?). Then,
having also identified the items to be ranked, the second box shows the pair ranking of each item against
each of the selected criteria, the multiplication of the number of votes by the criteria’s weight, the total for
each item and the final ranking. A sanity check needs to be done before accepting that the man-portable
bomb is really the best method for conducting a terrorist attack.
61
Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006,
pp 63-67, and the USEUCOM CTSA Training Package.
Ranking the Major Criteria
Votes Ranking Weight
Physical Destruction IIII (4) 3 .27
Political Disruption II (2) 4
Social Disruption I (1) 6
Economic Disruption II (2) 5
Ease of Delivery IIIII I (6) 1 .4
Ease of Acquisition IIIII (5) 2 .33
Cultural Impact I (1) 7
Selected Criteria
Items to be Ranked
Ease of
Delivery:
.4
Ease of
Acquisition:
.33
Physical
Destruction:
.27
Total Final
Ranking
Dirty Bomb 3x.4=1.2 2x.33=0.7 6x.27=1.6 3.5 7
Poison Gas 2x.4=0.8 5x.33=1.65 5x.27=1.55 4.0 6
Car Bomb 5x.4=2.0 6x.33=2.0 2x.27=0.5 4.5 3
Crash Airplane 4x.4=1.6 4x.33=1.3 4x.27=1.1 4.0 5
Blow up Airplane 6x.4=2.4 3x.33= 1.0 3x.27=.8 4.2 4
Man-Portable Bomb 8x.4=3.2 8x.33=2.6 1x.27=0.3 6.1 1
Suicide Bomb 7x.4=2.8 7x.33=2.3 0x.27=0 5.1 2
Biological Attack 1x.4=0.4 1x.33=0.3 7x.27=1.9 2.6 8
Nuclear Bomb 0x.4=0.0 0x.33=0.0 8x.27=2.2 2.0 9
Page 80 of 113
Compare
Change Analysis.62
Purpose: Change Analysis is a technique for gaining insight into the impact of incremental change over
extended periods of time that may otherwise go unnoticed.
Description: Incremental change can often be difficult to notice, and while the aggregate of the change
over time may be large, each actual change may be too small to attract notice. Change Analysis is a
process whereby modifications in the criteria or factors that encompass or define an issue or relationship
are studied to determine whether or not alterations have occurred over time.
Process: As with many aspects of intelligence analysis, the first and most important step in undertaking a
Change Analysis is the selection of the issue and the timeframe to be analysed. The issue can be virtually
anything for which defining criteria or factors can be identified (the state of an international relationship;
the influence of an insurgent; etc.), and the timeframe is what ever is appropriate. Once the issue and the
time have been determined, then:
Select the comparison criteria or factors - Determine the criteria or factors that will give the best
insight into the most telling implications of the change. The best criteria or factors will:
o Be observable, measurable or determinable at each interval.
o Give valid and reliable insight into the amount of change occurring.
o Remain stable over the timeframe of the event or relationship.
o Taken together, provide unique insight into the change in the event or relationship being
assessed. They should imply one conclusion and only one. If other interpretations and/or
hypotheses are plausible, additional or more diagnostic indicators should be identified.
Establish appropriate time intervals – The timeframe, determined initially along with the issue,
needs to be broken into discrete segments that will illuminate the amount and speed of change
taking place. These segments should:
o Begin at a logical start point for the phenomena to be assessed.
o Be consistent (monthly, yearly, quarterly, etc.).
o Be tied to watershed events, such as elections, or recurring crises.
o Facilitate the measurement of the comparison criteria or factors.
Record the status of each criterion or factor at each interval – Determine the state of each
criterion or factor at the end of each interval, using a measurement process that is consistent in
detail and focus throughout.
Assess the change – Examine both the amount of change between the time intervals and the
overall change across the entire timeframe. Identify and assess the implications of:
- The rate of change for each criterion between time intervals and for the entire timeframe.
- The correlation between external events and the degree of change observed (did an
external event cause less or greater change compared to other periods?).
- Turning points that have been identified – and their implication for the future.
- Continued change at the current rate.
Weakness: Using selection of poor or inappropriate comparison criteria, factors, timeframe, or intervals
can cause misleading results and even a failure to see important changes.
62
Developed from the “Change Analysis” structured analytic technique authored by Jay Hillmer, DIA, Oct 2010.
Page 81 of 113
Generate Scenarios
Cone of Plausibility.63
Purpose: To identify drivers that shape current events in a subject area, and then generate a series of
plausible scenarios that are expected to describe the endpoint situation.
Description: The Cone of Plausibility is a disciplined scenario generation technique that helps analysts
imagine plausible futures and their effects. It is valuable in understanding the drivers that shape current
events and as a form of Strategic Warning.
Process: The Cone of Plausibility process begins with a very careful definition of the issue to be
examined. This will generally be in the form of - What will ____ look like in __?, where the second
____ is a specific time (be it two weeks, two
months, five years, etc.) .
Once the question and time frame have been
decided upon, the analyst needs to first identify and
then describe the drivers that are most useful in
defining the issue. It is important both that the
drivers and their descriptions accurately portray the
current situation, and that they be expected to
generally remain valid throughout the period.
Then, with a firm and thorough understanding of the
issue at hand, encapsulated in the drivers and
descriptions, the analyst can go on to assemble a Baseline Scenario. This scenario assumes that the
drivers and their descriptions will remain valid throughout the period. Next, a Plausible Scenario can be
produced by changing one (or more? – but be careful) driver and description – usually the one that is least
likely to hold true. Finally, a Wildcard (or another Plausible) Scenario can be generated by changing a
different baseline driver and description. Typically, the wildcard describes a future that is low probability
but high impact, or it may be the most dangerous scenario.
Requirements: Analysts involved in scenario generation through the Cone of Plausibility technique need
to have a thorough understanding of the subject under study, need to be able to select and define the
drivers that are most likely to remain valid throughout the timeframe selected, and need to be able to
imagine future impacts.
Strengths: The Cone of Plausibility Technique is particularly useful in providing decision makers with
discussion material, both through the drivers that define the subject area and the generated scenarios.
63
Developed from UK DIS materials.
Now ?
DriversDescriptions
Scenario 2:Plausible
Scenario 1:Baseline
Scenario 3:Wild Card
Now ?Now ?
DriversDescriptions
Scenario 2:Plausible
Scenario 1:Baseline
Scenario 3:Wild Card
Now ?Now ?
DriversDescriptions
Scenario 2:Plausible
Scenario 1:Baseline
Scenario 3:Wild Card
Now ?Now ?
DriversDescriptions
Scenario 2:Plausible
Scenario 1:Baseline
Scenario 3:Wild Card
Page 82 of 113
Generate Scenarios
Quadrant Hypothesis Generation Technique.64
Purpose: A rapid method of generating four hypotheses based on the two main factors, criteria or drivers
affecting the topic or event.
Description: For use when a topic or event has only two key driving forces that are easily identified and
with a wide consensus.
Process: The technique should be undertaken as
follows:
First, identify the two key factors (criteria or
drivers) and the descriptions of their
extremities through techniques such as
brainstorming or surveying experts. (If there
are more than two factors, use the Alternative
Futures Analysis technique – page 83.)
Then, draw a quadrant with the two factors
on the horizontal and vertical axes. (Public
Attitudes and Government Policy in the
example.)
Determine and place the extremes of the
factors at the end of each axis. (Outreach and
Crackdown for Government Policy, Anti-
Muslim Sentiment and Sympathy & Support
for Public Attitudes in the example.)
Finally, in each quadrant, formulate the
hypothesis that best leads to the end state that
is suggested by the combination of the
applicable extremes of the two drivers.
Strengths: The technique is especially good for developing future outcomes, in the form of four
hypotheses, from the interaction which the main forces either have or could have on the outcome.
Discussion of what comprises the two main factors (criteria or drivers), and identifying their extremes,
can be a useful exercise in itself.
Weaknesses: The technique depends upon the correct identification of two major driving forces. Where
there are numerous major driving factors, or there are significant disagreements on which forces are major
drivers, the technique should not be counted on.
64
Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006,
pp 25-26
Pub
lic
Att
itud
es
Topic: Home Grown Terrorism
Key Drivers:
Government Policy
Public Attitudes
Anti-Muslim
Sentiment
Sympathy
& Support
Government PolicyOutreach Crackdown
Hypothesis #1
Hypothesis #4
Hypothesis #2
Hypothesis #3
Pub
lic
Att
itud
es
Topic: Home Grown Terrorism
Key Drivers:
Government Policy
Public Attitudes
Anti-Muslim
Sentiment
Sympathy
& Support
Government PolicyOutreach Crackdown
Hypothesis #1
Hypothesis #4
Hypothesis #2
Hypothesis #3
Page 83 of 113
Generate Scenarios
Alternative Futures Analysis.
Purpose: To systematically identify a range of alternative futures, monitor for signposts, and thereby
avoid surprise.
Description: Alternative futures Analysis takes over when the
two factor limit of the Quadrant Hypothesis Generation (see
page 82) technique is too restrictive. In this technique, the
group of experts collectively identifies three (or more) factors,
criteria or drivers that are likely to shape an issue.
Then, during the “quadrant crunching” phase, each of the
factors is compared to each of the other factors in a 2x2 matrix,
producing a different hypotheses, “future world” or scenario in
each quadrant, as shown.
Strengths: Most other efforts to anticipate future events usually end up
with minor variations of current trends. The Alternative Futures
technique forces analysts to look at not only the most likely scenarios
but also those that might seem unlikely, even counter-intuitive. Some of
these may have high impact.
Weaknesses: This is a highly efficient and effective technique for
generating an extremely broad set of alternatives when faced with very
little data and high degrees of uncertainty, but,
it takes time and effort to do well, and
a diverse group of participants is critical.
Also, it is difficult to manage more than three factors. Four can be
combined six different ways producing 24 scenarios, six can be
combined 15 ways producing sixty scenarios, etc.
On Completion: Analysts can then ask: What signposts or indicators would exist on the path to each
“future world”?
Home Grown Terrorism
Key Drivers
A. Government Policy
B. Public Attitudes
C. Immigration Trends
A
B
A
C
B
C
Home Grown Terrorism
Key Drivers
A. Government Policy
B. Public Attitudes
C. Immigration Trends
A
B
A
C
B
C
A
B
A
C
B
C
Pub
lic
Att
itud
es
Topic: Home Grown Terrorism
Anti-Muslim
Sentiment
Sympathy
& Support
Government PolicyOutreach Crackdown
Valuing
Diversity
Touchy
Feely
Hard
Times
Necessary
Measures
Defining potential outcomes
Pub
lic
Att
itud
es
Topic: Home Grown Terrorism
Anti-Muslim
Sentiment
Sympathy
& Support
Government PolicyOutreach Crackdown
Valuing
Diversity
Touchy
Feely
Hard
Times
Necessary
Measures
Defining potential outcomes
Topic: Home Grown Terrorism
Select the scenarios deserving the most attention
Hypotheses
#1Hard
Times
Hypotheses
#4
Hypotheses
#3
Hypotheses
#5
Conflict
Brewing
Hypotheses
#7
Hypotheses
#9
Civil
Strife
Hypotheses
#11
A
B C C
A B
Hypotheses
#6
Hypotheses
#10
Legend – Deserving the most attention
Topic: Home Grown Terrorism
Select the scenarios deserving the most attention
Hypotheses
#1Hard
Times
Hypotheses
#4
Hypotheses
#3
Hypotheses
#5
Conflict
Brewing
Hypotheses
#7
Hypotheses
#9
Civil
Strife
Hypotheses
#11
A
B C C
A B
Hypotheses
#6
Hypotheses
#10
Legend – Deserving the most attention
Page 84 of 113
Understand Possibilities
Force Field Analysis.65
Purpose: To better understand forces or key factors, criteria or drivers influencing an issue or change. It
is also a useful tool for generating signposts and indicators.
Description: Force Field Analysis investigates the balance of forces acting on an issue or influencing a
change. It attempts to identify the most important players and target-groups involved in the issue and their
relative influence. It is excellent in providing a vehicle to promote discussion of a topic.
If a change is contemplated, Force Field
Analysis can help identify potential opponents
and allies. At the onset of a study, it clarifies
key factors or forces. It also helps challenge
assumptions about these factors and highlights
those that are stabilising or destabilising.
Process: To carry out a Force Field Analysis:
Describe, in detail, the plan or proposal
for change.
List all of the forces supporting the
change in one column.
List all of the forces resisting the
change in another.
Scores can be assigned to each force showing
relative strengths. These values can then be
used to compare relative weights of the forces
for verses the forces against the proposal. They
can also be used to show the effect of a change
in the influence of a factor, if it can be modified.
The “exam question” must be clearly identified and understood by the study team before embarking on
the identification of positive and negative forces.
Strengths: Force Field Analysis breaks down the core forces / drivers that influence or impact a
situation. It allows users to create, and then consider, a wider array of alternative scenarios and
possibilities. It is particularly useful when timelines are short, or when a relatively simple yes/no
assessment is required.
Weaknesses: The technique does not generate a specific “baseline” scenario and the forces identified as
impacting on the subject may be too numerous, overly complicating the process. When generating
scenarios, because of the polar nature of the forces, the gray areas may not be adequately examined.
Software: A Force Field Analysis Worksheet and instructions are available at
http://www.mindtools.com/rs/ForceField.
65
Developed from UK DIS material.
Forces for
change
Forces against
changeForces for
change
Forces against
change
Page 85 of 113
Understand Possibilities
Counterfactual Reasoning.66
Purpose: Counterfactual Reasoning is a structured process for generating alternate future scenarios that
are set in context.
Description: This technique is
particularly suited to answering
what-if questions where the
context includes: the different
situations which could bring those
scenarios about, and the other
events which will also be taking
place during the same time
period.
Process: Once the analytic
challenge has been defined (What
will B look like if A happens?),
the Counterfactual Reasoning
technique uses the following seven step process:
Generate antecedent scenarios – Identify the possible ways that A could come about.
Rank the antecedent scenarios – In this step we rank order the scenarios in terms of the
probability of each scenario occurring, and the length/complexity of each scenario (in other
words, the number of different sequences of events that need to converge to bring about the
scenario).
Generate intermediate events – Identify the relevant events that will independently occur
irrespective of A, and potentially influencing B.
Rank the intermediate events – In this step we need to examine the likelihood that the
intermediate events will be more or less important given the occurrence of event A. That is, does
the likelihood of the intermediate events being influential increase, decrease or remain the same if
A occurs?
Generate consequent scenarios – Identify potential outcomes (B) that are consistent with both the
antecedent scenarios and the intermediate states.
Rank the consequent scenarios – Determine whether or not the scenarios can have their
probability quantified (most likely/most dangerous/most costly/etc.). If they can, rank order them
– especially if there are lots of them.
Select the most significant consequent scenarios – Finally, identify the scenarios that are most
likely/most dangerous/most costly for inclusion in an analytic product.
Strengths: Using this technique to answer a “What will B look like if A happens?” question, prompts the
analyst to: make reference to the events that will likely cause A; incorporate the influence of events that
will take place irrespective of A but which may influence B; and put the consequences (B) in order of
probability.
66
Developed from Dr Noel Hendrickson’s presentation on counterfactual reasoning given to the 4-Eyes Analytic
Training Workshop, 12-16 Jan 2009 and Counterfactual Reasoning – A Basic Guide for Analysts, Strategists, and
Decision Makers, Dr Hendrickson, The Proteus Monograph Series Vol 2, Issue 5, Oct 2008.
Timeline
Today Future
Antecedents
Intermediates
Timeline
Today Future
Antecedents
Intermediates
Page 86 of 113
Understand Possibilities
Bow-Tie Technique. 67
Purpose: The Bow Tie Analytic Technique is used to model the causes and consequences of a specific
“undesirable event” and can help identify actions that have the best chance of reducing the most likely
causes and mitigating the worst consequences.
Description: This
technique is useful in
that it identifies both the
causes that can lead to
the undesirable event
and the consequences
emanating there from.
The technique also
allows the development
and consideration of
both proactive and
reactive control
measures, and can be
used to show the effect
of a policy, or physical
change on the issue.
Usage: The process begins by identifying, as specifically as possible, the “event” that is to be examined.
This can be a single undesirable event – such as the loss of control of a vehicle (see the example in the
boxes), or a failure of policy – the diplomatic or military eviction from a locale, or a political
development – the rise to power of a dictatorial leader. The event does not, however, need to be negative.
Equally plausible are desirable goals, such as valid elections or the achievement of an objective.
Once the event has been defined, analysts should identify causes that could bring this event about.
Causes can be thought of as factors or drivers – i.e. circumstances, facts or influences that, either
separately or in combination with each other could cause the event to take place. Depending on
the number of causes identified, a process of ranking them – by likelihood or impact – may be
necessary.
Then, analysts should identify the consequences of the undesirable event. These may take
numerous forms (physical, psychological, internal or external, etc.) and it may be necessary to
separate them into categories or rank them in order of priority or impact.
Having established the elements of the “bow tie,” analysts should
begin to consider the impact of both proactive and reactive
control measures on the issue. In this example, three categories
have been used, physical or engineered, maintenance, and
procedural. Maintenance, in this regard engenders a combination
of physical or engineered activity or presence, with a policy
statement. This is, however, only one approach to categorising
the controls and the groupings in the environmental scanning techniques may actually be more
suited to this problem being studied.
67
Developed from a briefing provided by Mike Standbrook, of BC Hydro, to CAPIA, on 23 Sep 2009. (According
to Mike Standbrook, Royal Dutch Shell used a similar process 20+ years earlier).
The
Event
Consequence ACause 1
Consequence B
Consequence C
Consequence D
Cause 2
Cause 3
Cause 4
Consequence E
Consequence F
Proactive Controls Reactive Controls
Elimination
of causes
Reduction of
probability of
undesirable event
Reduction of
consequences
Recovery from
consequences
Undesirable Event
The
Event
Consequence ACause 1
Consequence B
Consequence C
Consequence D
Cause 2
Cause 3
Cause 4
Consequence E
Consequence F
Proactive Controls Reactive Controls
Elimination
of causes
Reduction of
probability of
undesirable event
Reduction of
consequences
Recovery from
consequences
Undesirable Event
Physical or Engineered
Maintenance
Procedural
Controls:
Physical or Engineered
Maintenance
Procedural
Controls:
Page 87 of 113
Probability
Co
ns
eq
ue
nces
Pre-Policy
Post-Policy
Probability
Co
ns
eq
ue
nces
Pre-Policy
Post-Policy
Proactive controls are considered
in two ways. First, the
elimination of causes category
sets out to find ways, physical,
maintenance or policy related,
that will eliminate the existence
of the cause. (If an elimination,
or even amelioration, can be
effected, perhaps the undesirable
event will not take place.)
Second, the proactive control will
consider ways that the probability
of the undesirable event can be
reduced. The controls identified
in the first category are almost
certain to be specific to one
cause, while the controls
identified in the second category
may be associated with one or
more causes.
Reactive controls are also
considered in two ways. First
are physical or policy initiatives
that will reduce the potential
impact of the consequences or
cause the shift from a more to a
less severe consequence.
Second, are initiatives that will assist in the recovery from the consequences. It should be noted
that reactive initiatives are unlikely to be consequence specific.
Finally, if useful, analysts can consider the impact that a change in
either the physical or policy realm may provoke. This can be shown
in many ways, including using the construct in the box on the right,
where the probability of each of the consequences being manifest,
pre-change, is shown dashed, and the probability of each
consequence being manifest post-change is shown solid.
Loss of
Control of
Vehicle
Human error
Slippery road
conditions
Vehicle malfunction
Sudden road
obstruction
Cause Undesirable Event
Don’t drive
when roads
are slippery
Proactive Controls
Elimination
of causes
Reduction of
probability of
undesirable event
Gravel roads
when slippery
Construct roads
underground
Lower speed
limit
Driver
training
4x4
Capability
Periodic vehicle
check-ups
Appropriate
Tires
Walk don’t
drive policy
New vehicles
Regular vehicle
maintenance
Loss of
Control of
Vehicle
Human error
Slippery road
conditions
Vehicle malfunction
Sudden road
obstruction
Cause Undesirable Event
Don’t drive
when roads
are slippery
Proactive Controls
Elimination
of causes
Reduction of
probability of
undesirable event
Gravel roads
when slippery
Construct roads
underground
Lower speed
limit
Driver
training
4x4
Capability
Periodic vehicle
check-ups
Appropriate
Tires
Walk don’t
drive policy
New vehicles
Regular vehicle
maintenance
Reactive Controls
Reduction of
consequences
Recovery from
consequences
Undesirable Event
Loss of
Control of
Vehicle
Consequences
Minor Injuries
and/or Damage
Life
Threatening
Injury
Fatality
No Injuries or
Damage
ABS Brake
System
Air Bags
Mandatory
seat belt
use
911
Emergency
Response
OnStar
First-Aid
Supplies
on Board
Reactive Controls
Reduction of
consequences
Recovery from
consequences
Undesirable Event
Loss of
Control of
Vehicle
Consequences
Minor Injuries
and/or Damage
Life
Threatening
Injury
Fatality
No Injuries or
Damage
ABS Brake
System
Air Bags
Mandatory
seat belt
use
911
Emergency
Response
OnStar
First-Aid
Supplies
on Board
Page 88 of 113
Demonstrate
What If? Analysis.68
Purpose: What If? Analysis can be a useful technique for developing the “analytic proof” that may help
clients understand how a particular event could happen, even if that event seems unlikely at the present
time. Note: The UK-developed Backcasting-Light (Quick I&W) process is very similar.69
Description: By shifting the focus from whether the event could occur to how it may happen, What If?
Analysis encourages analysts to suspend judgement about the likelihood of the event and focus more on
what developments – even unlikely ones – might enable such an outcome. The technique shifts
discussion from “Could it happen?” to:
How much does it matter?
How is it most likely to come about?
Has the probability of the event happening changed?
The technique also gives decision makers:
A better sense of what they might be able to do today to either prevent an untoward development
from occurring or to leverage an opportunity for advancing their interests.
A specific list of indicators to monitor to see if developments point to the event actually
occurring.
Process: Individuals or groups can begin a What If? Analysis by either clearly stating the conventional
analytic line and then stepping back to consider what alternative outcomes are too important to dismiss,
even if unlikely. Or, by simply pretending that what could happen has already occurred. It is important
to be precise in defining both the event and its impact; sometimes it may even be helpful to posit a
triggering event.
Once the endpoint (that is the event and its impact) has been established, analysts or teams should:
Develop a chain of argumentation – based as much on logic as evidence – to explain how this
event could have come about.
Work backwards from the event in concrete ways – specifying what must actually occur at each
stage of the scenario.
Identify one or more plausible pathways or scenarios to the unlikely event; very often more than
one will appear possible.
Generate a list of indicators or “observables” for each scenario that would point to the events
starting to play out.
Assess the level of damage or disruption for a negative scenario and how difficult it would be to
overcome.
Assess the overall impact of a positive scenario and how best it could be enabled.
Rank the scenarios in terms of which deserve the most attention by taking into consideration the
difficulty of implementation and the potential impact.
Monitor the indicators on a periodic basis.
68
Developed from the CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence
Analysis, Jun 2005, pp 26-27, and the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.35.
69
Quick Wins for Busy Analysts, Jul 2012, pp 32-33.
Page 89 of 113
Demonstrate
Chronologies and Timelines.70
Purpose: By providing a method of displaying data over time, chronologies and timelines provide a
visual way to look for relationships and patterns in the data connecting persons, places, organisations,
phone numbers, and other activities.
Description: Chronologies list events in the order
they occurred, usually in narrative or bulleted form.
This allows the analyst to identify gaps or
unexplained time periods, consider the implications
of the absence of evidence, and draw conclusions or
make recommendations.
Timelines arrange information graphically along a
chronological spectrum. Information can be
presented visually by categorising the data and
displaying it above and below the line. This allows
the analyst to look for links between timelines of activities of persons or organisations, identify gaps or
unexplained time periods, and consider the implications of
the absence of evidence.
Chronologies and timelines are useful throughout the
analytic process for structuring and visualising data. At the
beginning of the analysis, they can be used to sort the data
by time period and to visually display time series data.
During the analytic process, newly acquired data can be
added to chronologies and timelines as events happen or
when additional information comes to light. In the final
product, chronologies and timelines are particularly useful for displaying the data for clients.
Strengths: Chronologies and
timelines help in organising and
sorting information over time or into
designated time periods. Once the
data is structured, the analyst can more
efficiently examine relationships
among the various items noted. The
analyst can quickly see the flow of the
data, identify relationships, discover
gaps and generate requirements for
additional research or investigation,
and present results to clients.
70
Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.9, Graphic from the DIA’s A
Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006, p 38
A chronology is usually a simple list of events, typically from left to right.
Who is giving the orders?
Page 90 of 113
Intelligence
Analysis
Alternate
Explanations
Not
Data dumpKey
Assumptions
Evidence
BLUF
Support
inferences
Exec
Summaries
Amplify in
subsequent
Audience
read
initials
Important
points
Position in
para lead
sentences
Summary in
lead para
Para lead
sentences
Selected
Relevant
Sourced
Footnoted
Customer
need
Identify
indicators
Deal with
carefullyHighlight
Integral
Taken for
granted
Against
For
Highlight
gaps
Provide
credibility
Single
cannot
Seldom
black & white
Imperfect
knowledge
Intel
customers
Want
insight
Well
read
Intelligence
Analysis
Alternate
Explanations
Not
Data dumpKey
Assumptions
Evidence
BLUF
Support
inferences
Exec
Summaries
Amplify in
subsequent
Audience
read
initials
Important
points
Position in
para lead
sentences
Summary in
lead para
Para lead
sentences
Selected
Relevant
Sourced
Footnoted
Customer
need
Identify
indicators
Deal with
carefullyHighlight
Integral
Taken for
granted
Against
For
Highlight
gaps
Provide
credibility
Single
cannot
Seldom
black & white
Imperfect
knowledge
Intel
customers
Want
insight
Well
read
Demonstrate
Mind Mapping.
Purpose: To improve critical and creative thinking by depicting relationships visually.
Description: Analysts often start collecting information without sufficient consideration of the problem
or question they are facing. Whether what is needed is a conceptual framework or a collection plan, a
mind map can better illuminate the path ahead.
The four basic steps to
creating a mind map:
Write your topic
in the centre of a
large piece of
paper;
From the central
topic, radiate out
key words and
the most
important ideas
you have about
the topic, each on
a separate, thick
line;
Branch thinner
lines off the ends
of the appropriate
main lines, to
show supporting data (the more important the data, the closer it should be to the central topic or
idea); and
Use images and colors freely in your own special code to show people, topics, themes,
associations or dates, and to make the Mind Map more memorable.
Mind maps are well-suited to brainstorming because they don’t impose linear thought (like a hierarchical
“table of contents” approach does). One thought often spawns another, sometimes connecting things in
unexpected ways.
Mind maps can be created by hand using pencil and paper, on the web (Comapping, Mindomo,
MindMeister, and bubbl.us), or with software (ConceptDraw, FreeMind, TopicScape, iMindMap,
PersonalBrain, or MindManager).
Strengths: Mind maps are simple to create. They are intuitive, reflecting the way you think. They
visually capture, organize and communicate ideas and information effectively. Users have greater
retention and recall over traditional text-based approaches. They provide a single view of all the aspects
of a project. Lots of information can be taken in at a glance.
Note: The mind map in the box above uses the material in the Presentation of Analysis section on page
100.
Page 91 of 113
Demonstrate
Matrices.71
Purpose: This technique produces a grid with as many cells as required to sort data and gain insight.
Description: Matrices are useful whenever there are more options or more intricate data than can be
conceptualized at one time without a visual representation. Whenever information can be reduced to a
matrix, it provides analytic insights.
Process: Matrices can be rectangular, square, or triangular depending on the purpose and number of rows
and columns required to enter the data.
Step One: Draw a matrix with sufficient columns and rows to enter the two sets of data to be
compared.
Step Two: Enter the range of data or decision criteria along the horizontal axis (first column) and
along the vertical axis (first row).
Step Three: In the grid squares in between, note the relationships or lack thereof in the cell at the
intersection between the two associated data points.
Step Four: Review the hypotheses developed for the issue in light of the relationships shown in
the matrix and, if appropriate, develop a new hypothesis(es) based on the insight gained from the
matrix.
Strengths: Matrices are
exceptionally useful in isolating
critical data when there is an
abundant amount of overall
information relevant to an issue.
When used to review data related to
options, such as the analysis of
competing hypotheses, it enables
analytic focus on each option,
improving comparison. Matrices allow elements of a problem to be separated and categorized by type, for
comparison of different types of information or of pieces of the same type of information. Matrices also
allow analysts to identify patterns or
correlations within the information.
Weaknesses: The two-dimensional
design of matrices limits their use
for collating data on complex issues.
Leaving out pertinent data easily
oversimplifies an issue.
71
Developed from the DIA’s A Tradecraft Primer: Basic Structured Analytic Techniques, First Edition, Mar 2006,
pp 41-42. Second graphic – Adversarial Options Matrix from DIA CTSA Training Package, 2008.
Options Reasons for
Adopting
Reasons to
Reject Adoption
Intended
Results of Adopting
Implications of
Adopting
Indicators
that Option Adopted
Page 92 of 113
Demonstrate
Decision Trees.
Purpose: To establish chains of decisions and/or events which illustrate a comprehensive range of
possible future actions.
Description: Decision trees chart the range of options for a given decision point, give estimates of value
or probability for each option, and show possible outcomes.
Decision trees are useful to aid our
own decision making by explicitly
comparing options, or to create a
model of our subjects’ decision
making and possible actions.
A decision tree illustrates a
comprehensive set of possible
decisions and the possible outcomes
of those decisions. It can be used to
assess the probability of any given
sequence of decisions.
A decision tree is structurally
similar to critical path analysis and to program evaluation and review technique (PERT) charts. However,
both these techniques show only the activities and connections which need to be undertaken to complete a
complex task. A timeline analysis as done in support of a criminal investigation is essentially a decision
tree drawn after the fact, showing only the
paths taken.
Analysts producing a decision tree to model
a subject’s decision making need to have a
rich understanding of the subjective and
objective worlds of the subject(s), including:
cultural norms and values, motives,
operating doctrine, operational environment,
sensitivity to risk, and other relevant factors.
Conditional Probabilities: Decision trees
can be particularly useful for analysts in
determining the conditional probabilities of
an event – as shown in the box on the right.
Strengths: Decision trees are simple to understand and interpret. A decision tree can be generated by a
group through brain-storming, and can also be posted for addition & comment over a period of time.
Both approaches can increase the completeness of the set of possible options and consequences.
Weaknesses: Relies on the accuracy of data used, completeness of the range of options assessed, and on
the validity of the qualitative probabilities/values estimated for each option. Also, a detailed decision tree
can present the misleading impression that all possible options and/or outcomes have been considered.
Decision points are shown as squares,
and circles represent possible outcomes.
Page 93 of 113
Demonstrate
High Impact - Low Probability.72 Purpose: High Impact-Low Probability is a communication technique designed to convince a decision
maker that a seemingly unlikely event, that would have major consequences, might actually occur.
Description: High Impact-Low Probability analysis should be used when clients or intelligence staffs
need to be alerted to the potential that a seemingly long-shot development poses a credible threat.
Process: If the considered opinion is that an event is unlikely:
Clearly describe the unlikely event.
Define the high impact outcome of the event occurring as precisely as possible. Consider both
the actual event and the secondary impacts.
Identify information or evidence that suggests that the unlikely event is already actually
occurring.
Postulate additional triggers that would propel events in this unlikely direction, or factors that
would greatly accelerate timetables.
Develop one or more plausible pathways that would explain how this unlikely event could unfold.
Focus on the specifics of what must happen at each stage of the process.
Generate a list of indicators or signposts that would help identify that events were beginning to
unfold in this way.
Identify factors that could deflect this bad outcome or encourage a positive one (an opportunity
analysis).
Once the lists of indicators are developed, analysts must periodically review and update them as
necessary.
Strengths: High Impact-Low Probability analysis counters the natural tendency to focus attention on only
the most probable scenarios. This format also allows analysts to explore the consequences of an event –
particularly one not deemed likely by conventional wisdom – without having to challenge the main-line
judgement or to argue with others about how likely an event is to happen.
Mapping out the course of an unlikely, yet plausible, event can uncover hidden relationships between key
factors and assumptions; it also can alert analysts to oversights in the mainstream analytic line.73
Convincing others, particularly decision makers concerned about putting out today’s fires, that they
should devote time to considering a remote possibility for which there is currently weak evidence
requires: an authoritative or credible author; plausible arguments with compelling scenarios and open-
minded readers who are receptive to unconventional ideas. Absent some or all of those criteria, the High
Impact-Low Probability technique provides a process that, once the reader’s attention has been captured
by the clear description of the event and its consequent impacts, will lead the reader through a logical
progression designed to cater for and overcome the cognitive hurdles that the human mind is prone to
erecting.
72
Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, p.36 73
CIA A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis, Jun 2005, p 24
Page 94 of 113
Review Pre-Publication
Dialectic Inquiry.74 Purpose: Dialectical Inquiry challenges and disputes the assumptions and evidence used to frame and
support an existing analytic position to arrive at plausible alternative explanations through discussion.
Description: Dialectic Inquiry, like Devil’s Advocacy, uses intra-group conflict as the basis for solving
problems. However, it goes beyond merely finding flaws. Through constructive discussion, it attempts to
produce plausible alternative analysis. This technique is a valuable way to counter mindsets and bias
through the critical, but collegial, examination of existing analytic positions.
Process: Typically the dialectic process begins with a
written product, perhaps even a draft, where the
analytic position is identified and supporting evidence
and assumptions are clearly stated.
A team of analysts then examines the product for
evidence and assumptions, trying to imagine plausible
alternative interpretations of the assumptions and
evidence used. The team then seeks additional,
unused, evidence or assumptions that may contradict the stated analytic position and produce an
alternative interpretation.
Throughout this process, the key information, assumptions and their alternative interpretations are
examined, exposing any flaws. After discussing and reconciling any disagreements in interpretation, the
group agrees on which information and intelligence has
diagnostic value, and which assumptions are still valid. Finally,
the team may, if appropriate amend the initial analytic position
based on surviving information, intelligence and assumptions.
Strengths: By attacking and challenging assumptions and
evidence, the dialectic inquiry technique counters bias and helps
with prediction. If done thoroughly, only the best solutions will
survive critical examination and all flaws in the argument will
be detected. The technique maximises the evaluation of sources and information and enables confidence
levels to be expressed more accurately.
Weaknesses: Dialectic examination does not guarantee a better assessment. There is always the
possibility that the team will bring bias and mindsets to the table. Analysts need to be able to step outside
of their analytic comfort zone and to criticize their existing assumptions, biases and mindsets, and to
consider unused assumptions and evidence.
The technique stresses critical examination over group harmony.
Lastly, Dialectic Inquiry must be done by a team and it can take a lot time - particularly if the new
analytic position is subjected to further dialectic examination as purists would suggest is required.
74
Developed from UK DIS material.
During Dialectic Inquiry, lists are
produced of elements that could be used
against the initial analytic position:
Evidence Used
Assumptions Used
Information or Intelligence Not Used
Assumptions Not Considered
Throughout the process, ask:
If this evidence or assumption
turns out to be false, what is the
impact on the analysis?
Page 95 of 113
Review Pre-Publication
Pre-Mortem Assessment.75 Purpose: The Pre-Mortem Assessment is a systematic assessment of how a key analytic judgement,
decision, or plan of action could go spectacularly wrong. It is, however, conducted prior to finalising an
analytic judgement or decision. The Pre-Mortem Assessment asks an analyst, or team of analysts, to
imagine that a report has gone forward, and now, several months or years later, it is learned that the
analytic judgement was flat wrong or the project failed spectacularly. The objective, now, is to figure out
how it could have failed.
Process: Analysts should begin by employing a structured brainstorming, or similar technique, to elicit
plausible explanations for the unexpected outcome. They should then develop a list of indicators that
would provide early warning that events are not proceeding as anticipated. Finally, a comprehensive
assessment of all possible sources of analytic error can be assembled by considering:
Contradictory or Anomalous Evidence – try to imagine how rejected or ignored data might be a
key clue to an alternative hypothesis. Is there any evidence that you rejected as unimportant
because it didn’t appear relevant, or you didn’t believe it?
Absence of Information – are there major information gaps? Did the absence of information
influence how the team arrived at its conclusion? Were there unusual information gaps?
Key Assumptions – could any be unsubstantiated, or need to be better caveated. What is the
impact of an invalid assumption? What would the impact be on the bottom line?
Credibility of Critical Evidence – if any evidence, or stream of reporting, turns out to be
incorrect, what impact will this have on the analysis? Is key evidence dated? Could sources turn
out to be unreliable? How credible was the critical evidence? Was a single source, INT, or
stream of information relied upon?
Deception – does anyone have motive,
opportunity, or means to deceive you or
provide misleading information? Is
there a past history of deception?
Finally, the possibility of common analytic
pitfalls should be considered. Were alternative
hypotheses considered? Might a change in the
broad environment (technological change,
globalisation, environmental change, etc.) have
an impact on the analysis? Etc.
Change of Frame: To symbolise the different role analysts are playing in this process, meetings should
be scheduled exclusively for this purpose, be led by a different person, and preferably should be held at a
different location. An experienced facilitator may be helpful.
This change in the frame of reference changes the group dynamics. The critical perspective will always
generate more critical ideas. Team members who may have previously suppressed questions or doubts
because they lacked confidence or wanted to be good team players are now empowered to express those
divergent thoughts. If this change in perspective is handled well, each team member will know that they
win points with their colleagues for being critical of the previous judgement, not for supporting it.
75
Developed from the Handbook of Analytic Tools & Techniques, Pherson, 2008, pp.31-32, and a draft version of
Richards J. Heuer, Jr. and Randolph H Person’s Structured Analytic Techniques for Intelligence Analysts, Mar
2009.
What if my main conclusion or key
judgement turns out to be flat wrong?
Did I ignore
contradictory
evidence?
What should
have the absence
of evidence
told me?
Were my
key assumptions
valid?
Did
deception go
undetected?
How reliable
was my key
evidence?
What if my main conclusion or key
judgement turns out to be flat wrong?
Did I ignore
contradictory
evidence?
What should
have the absence
of evidence
told me?
Were my
key assumptions
valid?
Did
deception go
undetected?
How reliable
was my key
evidence?
Did I ignore
contradictory
evidence?
What should
have the absence
of evidence
told me?
Were my
key assumptions
valid?
Did
deception go
undetected?
How reliable
was my key
evidence?
Page 96 of 113
Matching the Techniques to the Tasks76
For the inexperienced analyst, the problem is often one of not knowing which structured analytic
technique to choose for the problem at hand. The following series of questions are intended to
encapsulate as much of the breadth of analytic issues as possible, and for each, to provide
guidance as to which technique would be appropriate. This is not the definitive selection
process, indeed expertise and familiarity with the subject issue will often provide better
guidance, but it is a place to start.
What is it you want to accomplish?
Do you want to define a project or issue?
o Delphi Method
o Environmental Scanning
o Nominal Group Technique
o Reframing the Question
o Structured Brainstorming
Are you just getting started, wanting to generate a list of important factors, criteria or
drivers?
o Environmental Scanning
o Outside-In Thinking
o Mind Mapping
o Structured Brainstorming
Do you have a list, and want to prioritize it?
o Weighted Rankings
o Structured Comparison
Do you want to examine or visualise relationships between factors, indicators, players,
etc.?
o Chronologies and Timelines
o Cross-Impact Matrix
o Decision Trees
o Force Field Analysis
o Link Charts
o Mind Mapping
o Social Network Analysis
o Structured Comparison
Do you want to make sense of a recent event, or assess the likely outcome of an evolving
situation?
o Alternative Futures Analysis
76
Developed from a draft version of Richards J. Heuer, Jr. and Randolph H. Phearson’s Structured Analytic
Techniques for intelligence Analysts, Mar 2009. (Published by CQ Press, 2011)
Page 97 of 113
o Hypothesis Generator
o Outside-In Thinking
o Quadrant Hypothesis Generation Technique
Do you want to monitor a situation or issue and avoid being surprised?
o Alternative Futures Analysis
o Bow-Tie Technique
o High Impact - Low Probability
o Indicators of Change
o Indicators Validator
Do you want to generate and test hypotheses?
o Analysis of Competing Hypotheses
o Hypotheses Generator
o Hypotheses Review
o Quadrant Hypothesis Generation Technique
Are you trying to foresee the future?
o Alternative Futures Analysis
o Cone of Plausibility
o Counterfactual Reasoning
o Delphi Method
o Structured Brainstorming
Are you concerned about the effect of assumptions, mindsets and biases in your
analysis?
o Analysis of Competing Hypotheses
o Data Diagnostic
o Dialectic Inquiry
o Delphi Method
o Devil’s Advocacy
o Key Assumptions Check
o Pre-Mortem Assessment
Do you want to see things from other perspectives?
o Cross-Impact Matrix
o Delphi Method
o Devil’s Advocacy
o Hypothesis Review Technique
o Outside-In Thinking
o Pre-Mortem Assessment
o Red Cell
o Red Team
o Reframing the Question
o Team A / Team B
Page 98 of 113
Do you need to create a visualisation of an issue?
o Chronologies and Timelines
o Cone of Plausibility
o Counterfactual Reasoning
o Decision Trees
o Force Field Analysis
o Link Charts
o Matrices
o Mind Mapping
o Structured Comparison
Do you need to contribute to the warning process?
o Cone of Plausibility
o Environmental Scanning
o High Impact - Low Probability
o Indicators of Change
o Indicators Validator
o Structured Brainstorming
o What If? Analysis
Page 99 of 113
Write Clear, Concise, and Client Focused Reports
CFINTCOM Analytic Product Standards77
While recognizing that there is a wide variety of analytic products and target audiences,
intelligence analysis is best served by having all analytic elements adhere to a common set of
standards. To this end, all analytic products should:
1. Make accurate judgements and assessments. Analysts should apply expertise, logic and
reasoning to make the most accurate judgements and assessments possible given the time and
information available and known information gaps. Where products are estimative, the
analysis should identify and correctly characterize the impact of key factors affecting
outcomes or situations.
2. Be clear. Analytical products should incorporate the Bottom Line Up Front structure, be
internally consistent, and use language and syntax that convey meaning unambiguously.
Graphics and images should be readily understandable and should illustrate, support, or
summarize key evidence or analytic judgements.
3. Be insightful, timely and relevant. Analytic products should provide timely information and
insight on issues relevant to the products' intended consumers and/or provide useful context,
warning, or opportunity analysis.
4. Properly expresses confidence or uncertainties in analytic judgements. Where appropriate,
analytic products should indicate both the level of confidence in analytic judgements and
explain the basis for ascribing it. Sources of uncertainty – including information gaps and
significant contrary reporting – should be noted and linked logically and consistently to
confidence levels in judgements.
5. Properly identify assumptions. Analytic products should explicitly identify the critical
assumptions on which the analysis is based and explain the implications for the analysis if
those assumptions prove to be incorrect.
6. Consider alternative hypotheses. Where appropriate, analytic products should identify and
explain the strengths and weaknesses of alternative hypotheses, viewpoints, or outcomes in
light of both available information and information gaps. Analytic products should explain
how alternatives are linked to key assumptions and/or assess the probability of each
alternative.
7. Identify indicators. As appropriate, analytic products should identify indicators that would:
enhance or reduce confidence or prompt revision of existing judgements; signal whether
assumptions are more or less likely to be correct; or help clarify which alternative hypothesis,
viewpoint, or outcome is more likely or is becoming more likely.
77
Developed from the US ODNI’s Rating Scale for Evaluating Analytic Tradecraft Standards, (2007 ?).
Page 100 of 113
8. Use structured analytic techniques. To the extent possible, analysis should incorporate
insights from the application of structured analytic technique(s) appropriate to the topic being
analyzed.
9. Properly describe the quality and reliability of evidence. Analytic products should explain
which evidence is key to analytic judgements and why. When appropriate, factors
significantly affecting the weighting that the analysis gives to available information, such as
denial and deception, access, motivation, relevance or other factors affecting the quality and
potential reliability of the information, should be included in the product.
10. Exhibit consistency of analysis over time, or highlight changes and explain rationale.
Analytic products should deliver a key message that is either consistent with previous
production on the topic from the same analytic element or, if the key analytic message has
changed, highlight the change and explain its rationale and implications.
Presentation of Analysis
Well-presented intelligence analysis saves intelligence consumers time by using the structure of
the Bottom Line Up Front (BLUF).78
Assessments should also provide supporting evidence for
any inferences being drawn, explain the influence of key assumptions, and, when appropriate,
offer alternative explanations or hypotheses.
BLUF requires that the important elements of an assessment be made obvious, first, by
summarizing them in the lead paragraph, and then by sequentially positioning them in the lead
sentences of the subsequent paragraphs. By doing this, the audience needs only to read the initial
sentences to gain an understanding of the material. Any amplifying items should, consequently,
be placed in the subsequent sentences of the paragraph. By using this structure, accurate
executive summaries can be produced by incorporating the lead sentences of each paragraph
either in whole or in part.
Assessments should always present evidence to support any inferences being drawn. Simply
providing a data dump of seemingly-relevant material is not the objective here. Rather, only
carefully selected, truly relevant, properly sourced and footnoted material should be presented.
The objective is not for the analyst to show how much he or she knows, but to provide the
material that the client needs to know. Inherent in the selection of material is a good knowledge
of the client’s needs.
Key assumptions also need to be carefully dealt with in intelligence analysis. To this end, the
assessment should highlight key elements that are being ‘taken for granted’ and upon which the
integrity of the analysis rests. Good analytic products will also identify indicators that can be
expected to show whether – or not – the stated assumptions continue to hold true. Intelligence
gaps should also be highlighted when appropriate.
Finally, alternative hypotheses or explanations can provide a level of credibility to an assessment
that a single-minded explanation cannot. Even the best analyst is unlikely to have perfect
78
Based on Communicating with Intelligence, by James S. Major, Scarecrow Press, 2008
Page 101 of 113
knowledge of any intelligence issue being examined, and, while it is tempting to be categorical
in assessment, issues are seldom black and white. Further, our intelligence clients are, in the
most part, intelligent, well-read individuals who are more interested in an insightful treatment of
a subject than they are of the opinion of an unknown analyst.
Language of Uncertainty
An important aspect of any intelligence analysis is an expression of the likeliness that the
assessed event or development will or will not occur.
While the box on the left, below, suggests a definite hierarchy to the probability or likeliness
terms that an analyst may choose, the box on the right shows just how broad the interpretation of
the same terms can be. As a consequence, the development of and strict adherence to a set of
terms and definitions is strongly advised. If this is not possible, then ensure that the terms to be
used are specifically defined in a place where the reader cannot help but see them.
Improbable
Beyond aReasonable
doubt
Certain
unlikely
Likely
Groundsto suspect
Possible
Probable
Groundsto believe
0 10 20 30 40 50 60 70 80 90 100
Improbable
Beyond aReasonable
doubt
Certain
unlikelyunlikely
LikelyLikely
Groundsto suspect
Possible
Groundsto suspect
Possible
ProbableProbable
Groundsto believeGroundsto believe
0 10 20 30 40 50 60 70 80 90 100
Likely
Probable
Improbable
unlikely
Groundsto suspect
Groundsto believe
Beyond aReasonable
doubt
Possible
Certain
Likely
Probable
Improbable
unlikely
Groundsto suspect
Groundsto believe
Beyond aReasonable
doubt
Possible
Certain
Page 102 of 113
Expressing Analytic Certainty79
Provide strong judgements
Intelligence analysis is all about making judgements on the basis of information at hand,
assessing the situation based on sound analytical techniques and methodologies, and
understanding the target based on observations of past patterns of activity and research.
Intelligence reporting does not usually take the form of conclusive direct evidence (if it did we
would only be collating and evaluating facts). Our senior clients recognise that assessments are
made based on the best information available at the time of the assessment. The harm of a
wrong call is likely to be less than the harm of being irrelevant through overly conservative,
qualified, or out of date/late assessments. Analysts could qualify judgements so much that they
can never be wrong—but calling everything possible adds little value for decision makers.
When we make a judgement, we should do so:
with as few qualifications as possible,
with enough supporting evidence and logic to give authority to the judgement, and
with clarification of any vulnerabilities in the assessment.
Analytic certainty is based on two distinct components: likelihood and confidence, which we
define as follows:
Likelihood—a statement that expresses the probability, that is, the chance than an event
or development will or will not happen.
Confidence—an expression of the strength of our assessment based primarily on our
depth of understanding of the issue, the persuasiveness of the reasoning (taking into
account gaps and assumptions) and the potential for deception.
Likelihood
The following figure shows the terms that we will use to express the various levels of likelihood.
These words are roughly associated with the scale of 0/10 through 5/10 to 10/10. These numbers
are not intended to express a percentage or other numerical interpretation, but are simply to
indicate likelihood. Something that is conceivable is likely to be manifest in three out of ten
instances, for example.
In instances where an exploratory analysis is being undertaken, or when we simply do not know,
we will use terms such as may, might, unable to assess and undetermined.
79
Based on DIA Tradecraft Note 01-15: Expressing Analytic Certainty, 5 Jan 2015.
Page 103 of 113
Analytic confidence
Certainty is unlikely to be achieved, because of the nature of the intelligence challenge. If the
required information could be relied on, came from unimpeachable sources and was readily
available, there would not be a requirement for intelligence in the first place. The analytic
confidence level must therefore be made as clear as possible to the clients.
Analytic confidence is not an expression of an analyst’s personal belief that a judgement is
correct, nor is it a measure of the likeliness that the event will occur in the future. Rather, it is an
expression of the maximum degree of confidence that can be justifiably assigned to a judgement,
based on three factors:
Evidence – the strength of the knowledge base, to include the quality of the evidence and
our depth of understanding about the issue;
Assumptions - the number and importance of assumptions used to fill information gaps;
and
Reasoning - the strength of the logic underpinning the argument, which encompasses the
number and strength of analytic inferences as well as the rigour of the analytic
methodology applied to the product.
Each successively higher level of analytic confidence in the graphic below indicates a greater
degree of commitment.
When it needs to be stated, and that will not be very often, analytic confidence will be
characterised as high, moderate or low. If we have complete confidence in something, we would
probably be stating the obvious, and if we have no confidence, we probably will not be writing
on the subject.
Page 104 of 113
When confidence is medium or low, this confidence needs to be made obvious in the narrative
(either integrated into the text or in a separate box) and the factors contributing to this lower-
than-normal confidence would need to be explored in some detail.
Footnotes
Almost every analytic product will benefit
from a thorough and consistent
identification of the sourcing information
for all significant and substantive
reporting or other information upon which
the product’s analytic judgements,
assessments, estimates, or confidence
levels depend. The documentation of
sources in footnotes or endnotes enhances
the credibility and transparency of
intelligence analysis enabling readers and
intelligence managers to better understand
the quantity and quality of information
underlying the analysis. Footnoting is
particularly useful in the pre-publication,
editorial and approval phases, and may be
crucial in a post-publication inquiry.
Endnotes or footnotes, intended to achieve this end, should contain the following five items of
information, when available:
the abbreviated classification of the material from this source that was used in the paper
(not necessarily the overall classification of the source document),
the source (agency, publication, individual, etc.),
the report number, serial number or message reference number,
the item or article title (abbreviated if necessary), and
the date.
Examples of Footnotes
Email DFAIT Williams, K, Nairobi (Cabnet), Burundi, 1
Mar 2007 (U)
Media Reports East Africa Standard (Nairobi), Kagame quits army
ahead of poll, 13 Aug 2003 (U)
BBC Monitoring, quoting Rwanda Radio, President
leaves for Visit to Belgium, 10 Mar 2004 (U)
Websites
EIU, Country Profile Nigeria, 31 Dec 2003 (U)
Page 105 of 113
The Reasoning Process
Reasoning, the act of forming conclusions, may be inductive, deductive, or abductive.
Deductive Reasoning.
Deductive reasoning80
is reasoning which uses deductive arguments to move from given
statements (premises) to conclusions, which must be true if the premises are true. An example of
deductive reasoning, given by Aristotle, is
All men are mortal. (major premise)
Socrates is a man. (minor premise)
Socrates is mortal. (conclusion)
Inductive Reasoning.
Induction,81
or inductive reasoning, is the process of reasoning in which the premises of an
argument are believed to support the conclusion but do not entail it; i.e. they do not ensure its
truth. Induction is a form of reasoning that makes generalizations based on individual instances.
It is used to ascribe properties or relations to types based on a number of observations or
experiences; or to formulate laws based on observations of patterns. The following are examples
of induction:
All observed crows have been black.
Therefore:
All crows are black.
This exemplifies the nature of induction: inducing the universal from the particular. However,
the conclusion is not certain. Unless we can systematically falsify the possibility of crows of
another colour, the statement (conclusion) may actually be false.
I always hang pictures on nails.
Therefore:
All pictures hang from nails.
Assuming the first statement to be true, this example is built on the certainty that "I always hang
pictures on nails" leading to the generalisation that "All pictures hang from nails". However, the
link between the premise and the inductive conclusion is weak. No reason exists to believe that
just because one person hangs pictures on nails that there are no other ways for pictures to be
hung, or that other people cannot do other things with pictures. Indeed, not all pictures are hung
from nails; moreover, not all pictures are hung.
Abductive Reasoning.
80
Wikipedia 81
Wikipedia
Page 106 of 113
Abduction,82
or inference to the best explanation, is a method of reasoning in which one chooses
the hypothesis that would, if true, best explain the relevant evidence. Abductive reasoning starts
from a set of accepted facts and infers their most likely, or best, explanations. Abduction often
involves both inductive and deductive arguments. However, as the conclusion in an abductive
argument does not follow with certainty from its premises it is best thought of as a form of
inductive reasoning. What separates abduction from the other forms of reasoning is an attempt to
favour one conclusion above others, by attempting to falsify alternative explanations or by
demonstrating the likelihood of the favoured conclusion, given a set of more or less disputable
assumptions.
Fallacies
Intelligence reports need to be firmly based on irrefutable logic. Some errors in reasoning or
flawed arguments occur so often that they are called fallacies, a few of the most common of
which are:
Inadequate sampling occurs when a sample, too small to represent an adequate measure,
is used to draw a conclusion.
Post Hoc ergo Propter Hoc occurs when it is assumed that if event B occurs after event
A, then A caused B.
Personification occurs when human characteristics are ascribed to objects or concepts.
Prevalent Proof occurs when mass opinion is used as a method of verification.
Appeal to Authority occurs when the opinion of a recognised expert is automatically
seen as valid.
Ad Hominem occurs when an argument is dismissed by attacking the person making the
argument.
False Dichotomy occurs when a set of possibilities is arbitrarily reduced to only two
opposing ones.
Non Sequitur refers to an argument in which the conclusion does not follow the premise;
a logical connection is implied where none exists.
Tautology is an argument that uses circular reasoning; the conclusion is its own premise.
Pre-ATIP Processing83
Pre-ATIP processing is meant to facilitate an efficient response to ATIP requests and keep track
of the responses that have already been provided. This process is especially important when
someone other than the author will have to respond to a request. It also allows the author to
indicate those passages that are particularly sensitive while the memory is still fresh.
For each paper, a table is prepared, indicating: ATIP number and request, or anticipatory;
subject/author; and severances.
Once the table has been filled out, it can be saved as an electronic file. If necessary, a hard-copy
can be printed and attached to the full document.
82
Wikipedia 83
This process is the initiative of George Betts, Global Division IAS.
Page 107 of 113
ATIP: ATI Request
A-2005-01111
Sea Slug notes
or (Anticipatory)
Subject/Author: IM -26/05 :Security Implications of Sea Slugs, 06 Dec 05;
Joe Smith
Overall Classification: Secret
Severances
Key Judgement 2 (S)
“Expansion of sea slugs pods...
Sever In Accordance With
Para 16(1) - Law Enforcement and
Investigations
Key Judgement 3 (S)
“Sea slugs are well established...”
Sever In Accordance With
Para 15(1) - Law Enforcement and
Investigations
Para 6, sentence beginning (S)
“Deep under the ocean .”
Sever In Accordance With
Para 13(1) - Information obtained in
confidence
Page 108 of 113
The Two-Hour Challenge
Users of this aide memoire can be forgiven if they observe that many of the techniques outlined
are best left to those with lots of time; i.e. the guys and gals that need to deal with the pressures
of producing a product every two months and who can be found staring out the window, for
hours on end, in quiet contemplation. If the conclusion is that only the prima dona strategic
analysts can possibly benefit from the Best Practices for Intelligence Analysis, it would be
wrong. Indeed, all analysts can reap significant benefit from the processes outlined in this aide
memoire.
Let’s consider that you have been asked to:
recommend a deployment route for impending operation Z,
explain what happened in location Y last night and the impact on Canada,
recommend a course of action to deal with an outbreak of X,
assess the utility of source W, or
assess the likely reaction of V to upcoming U.
Oh, by the way, you have two hours to do this in!!
So, what do you do? Running around with your hair on fire is one option, but perhaps not the
most efficient use of your time. What say you:
Start by figuring out what the question really is. In this regard, you may want to classify
the issue as a puzzle, mystery or mess. You may also want to paraphrase, broaden,
narrow or redirect the focus, ask “Why?” or turn the question 180 degrees in order to help
identify the real issue that needs to be addressed. You almost certainly also want to
determine which factors, drivers or criteria will have an important role to play in
explaining the issue.
Once you know what the question is and which factors will be important in answering it
for your client, gather the data you need. Given the two-hour time constraint, this likely
rules out asking for re-directed satellite coverage. Indeed, you probably want to focus on
your most trusted, readily available, sources of information. Keep track of the sources,
classifications and caveats on usage – you don’t need to create work for security
infraction investigators!
Then, think critically about the sources, quality and relevance of the information you
have available. Denial and deception, propaganda and exaggerations can all have a
negative impact on your analysis.
At this point – and not before!!! – develop multiple hypotheses or explanations. If only
one comes to mind, try harder. Do a quick quadrant hypothesis generation or try to do
some inside-out thinking. Remember, the paradox of expertise suggests that the more
you know about a subject area, the more likely you are to miss something.
Now is the time to consider the use of structured analytic techniques. Not all of them
take forever. Indeed, the diagnostic techniques like the Force Field Analysis,
Chronologies and Timelines, and Weighted Rankings can be very helpful in a short
period of time. They also produce graphical representations that may be useful in
educating/informing your client.
Finally, produce your report. Keep the bottom line up front and ensure that the
confidence you have in your judgements is made crystal clear. Include footnotes.
Page 109 of 113
Throughout this process, you should constantly challenge your assumptions and look for
the negative influence of bias. If you have time, ask someone in your network, or a
trusted outsider, to help.
Analysts almost certainly follow this process intuitively. The problem with the instinctive
method, however, is someday something may be omitted, and that momentary failure may result
in an assessment that gets someone inconvenienced or killed. A concerted effort to apply the
Best Practices may, on the other hand, help the analyst sleep better at night. A consistent
application of this process may also be handy if an analyst is forced to face an investigative
tribunal that is looking into an operational failure.
Judge: And how did you
come to make this
assessment?
You: As I am an internationally recognised expert in this field, I
simply drew on my experience to produce the assessment. The
short time constraint I was working under precluded any other
approach.
Or
You: Well sir, notwithstanding the short time constraints we were
working under:
- I took the time to ensure I understood both the type and
specifics of the question and the factors that would be
instrumental in answering it.
- I kept track of the data I gathered from the most
trustworthy and easily accessed sources, keeping track of
the distribution restrictions.
- I critically evaluated the source, content and relevance of
each piece of data.
- I developed multiple hypotheses.
- I utilised diagnostic techniques to help put the data in
context and produce representational graphics.
- Only then did I produce a report in which I placed the
bottom line up front and was very clear with my
confidence in the judgements I had included.
Throughout this process I was cognizant of the assumptions I was
working with and the potential for bias in my thinking. I
consulted both co-workers and my supervisor in an attempt to
mitigate the negative effects of both.
Page 110 of 113
Best Practices for Intelligence Analysis Review84
The Best Practices for Intelligence Analysts (BPfIA) Review is a systematic procedure that can
be undertaken by a small team or group can use to identify weaknesses in its own analysis (or by
a manager as a guide to help check on the analytic process undertaken by a team). The goal is to
confirm, or if necessary modify, both the analysis and the group’s overall confidence in its
judgements.
In order to extract the maximum benefit from this technique, all team or group members need to
don a hypothetical black hat85
and become critics rather than supporters of their own analysis.
From this re-framed position,86
they should critically review the analysis, answering the
following questions, in an attempt to generate new ideas.
Reflect on the Problem
o Should one expect to find a single correct answer, a most likely answer with one
or more alternatives, or a number of possible explanations for future
development?
o Is the appropriate question being answered, focusing on the factors that are
relevant to the client?
o Does the analysis deal with the W5H-SW?
o Does the analysis cater to the client’s needs (policy relevant, timely and
appropriately detailed)?
Collect Information
o Was a collection plan assembled?
o Were requirements identified?
o Were indicators identified and validated?
o How many different sources (INTs) were used?
Critically Evaluate Information
o For each important/critical piece of information:
What was the reliability and credibility was assigned to the provider?
What was the quality of the information itself?
What was the relevance of the information?
Is any evidence so dated that it may no longer be valid?
And what is the likelihood that deception was involved?
o Are there major information gaps?
84
Inspired by the Structured Self-Critique Technique that was developed by Richards Heuer and Randy Pherson and
appeared in the draft Chapter 9 Structured Analytic Techniques for Intelligence Analysts, Mar 2009. 85 The Black Hat is one element of Edward de Bono's "Six Thinking Hats" approach to decision-making. Using this
approach, the White hat provides facts and information, the Red hat injects feelings and emotions, the Yellow hat
provides positive judgements, the Green hat focuses on alternatives and learning, the Blue hat injects the big picture,
and the Black hat focuses on critical judgements. 86
A frame is any cognitive structure that guides the perception and interpretation of what one sees. An individual or
group can change their frame of reference by simply changing the questions it asks or changing the perspective from
which it asks them. This technique asks team members to reverse their role from advocate to critic in order to
explore potential weaknesses in the previous analysis. Effectiveness depends largely on how fully and
enthusiastically the group embraces the imaginative or alternative role they are playing. Just going through the
motions is of little value. (Extracted from the draft Chapter 9 of Heuer and Pherson’s Structured Analytic
Techniques for Intelligence Analysts, Mar 2009.
Page 111 of 113
Is this absence of information readily explainable?
o Is there any anomalous evidence that would have been important if it had been
believed or if it could have been related to the issue at hand?
o Is the issue relatively stable, or is the situation undergoing, or about to undergo,
significant change?
o If the information deals with decision-making, is the behaviour based on:
a rational actor model,
the result of bargaining between political or bureaucratic forces,
the result of standard organisational processes, or
the whim of an authoritarian leader?
o Does the team have the cultural expertise to judge the thought processes of the
culture being reported on?
Hypotheses
o What alternative hypotheses were considered?
What evidence was there for and against these (not-selected) hypotheses?
Are there reasonable alternative interpretations of the evidence that would
give greater credibility to a rejected hypothesis?
o Might a change in the broad environment (technological change, globalisation,
environmental change, etc.) have an impact on the analysis?
Assumptions, Mindsets and Biases
o What are the key assumptions inherent in this assessment?
How recent and reliable is the evidence supporting each of these
assumptions?
o Which one or two assumptions would have the greatest impact on the judgement
if they turned out to be wrong?
Collaboration
o Did the analytic team seek a broad range of diverse opinions by including analysts
from other offices, agencies, academia, or the private sector?
Structured Techniques
o What analytic techniques were used in the preparation of this assessment?
Are there other techniques that may have been more appropriate or which
may have achieved a different analytic result?
Communicate
o Is the logic in the assessment transparent? Can the logic trail be mindmapped or
otherwise demonstrated?
o Is the bottom line up front and obvious?
o Are all probability terms in keeping with the appropriate standard? If not, are the
non-standard usages made clear?
o Is the level of analytic confidence clearly stated?
Does the confidence level accurately reflect the group’s confidence in the
evidence that the analysis rests upon?
Does the confidence level accurately reflect the group’s confidence in the
analytic reasoning that informs the analysis?
Does the confidence level accurately reflect the group’s confidence that
the analysis was not compromised by deception?
Page 112 of 113
After answering each of these questions, the group should take off the black hats and consider
the appropriateness of both the assessment itself and the level of confidence expressed in the
assessment. Modifications should then be made if necessary.
The change in frame of reference gives the analytic group an opportunity to change its dynamics
and perhaps generate critical new ideas. Team members that have previously suppressed
questions or doubts because they lacked confidence or wanted to be good team players are now
empowered to express those divergent thoughts. If this change in perspective is handled well (a
facilitator may be required), each team member will know that they win points with their
colleagues for being critical of the previous judgement, not for supporting it.
The success of this process, however, depends in large measure on the team members’
willingness and ability to make the transition from supporters to critics of their own ideas. Some
analysts lack the intellectual flexibility to do this well.
Page 113 of 113
Analytic Rigour Map