1

Understanding and Improving Intelligence Analysis:

Learning from other Disciplines

12-13 July, 2012 * London, England * RAF Club

The purpose of this conference is to engage in a cross-disciplinary discussion about the value of learning

from other fields to improve both the understanding and the practice of intelligence analysis. It will also

create the network and infrastructure for an international research collaboration for the study of

intelligence analysis.

Intelligence, like journalism, involves the acquisition, evaluation, and dissemination of information. In

1949, Sherman Kent, described as the father of US intelligence analysis, said:

“Intelligence organizations must also have many of the qualities of those of our greatest metropolitan

newspapers. …They watch, report, summarize, and analyze. They have their foreign correspondents and

home staff…. They have their responsibilities for completeness and accuracy—with commensurately

greater penalties for omission and error. . . They even have the problem of editorial control….

Intelligence organizations (should) put more study upon newspaper organization and borrow those

phases of it which they require.”

But the similarities between intelligence analysis and journalism are not unique. Professionals in other

fields—including medicine, the social and behavioural sciences, history and historiography, anthropology

and other disciplines engaged in ethnographic research, econometric forecasting, and legal reasoning—

also face many similar challenges to those that exist in intelligence analysis, including:

Difficulties acquiring information from a wide variety of sources

Vetting and evaluating the information that is acquired

Deriving understanding and meaning from that information

Impact of deadlines, editing, and other production processes on accuracy of analysis and

assessment

Problems in dissemination and distribution to consumers or customers

Managing relationship between producer and consumer (role, responsibility, independence &

objectivity)

Developing professional infrastructure (recruit, select, train, & develop personnel; code of ethics)

Overcoming impact of changing technology and alternative information distribution systems

How do practitioners in various non-intelligence fields overcome these kinds of challenges? How are their

challenges similar to or different from those that exist in the intelligence arena? What can be learned from

the comparison?

This event has been funded through a grant from the Brunel University Research and Innovation Fund.

They are organized and hosted by Brunel University’s Centre for Intelligence and Security Studies in

collaboration with University of Mississippi’s Center for Intelligence and Security Studies.

2

Conference agenda Day 1: Thursday 12 July

8:00 Registration and Coffee

9:30 Welcome Part 1: Stephen Marrin (Brunel University): Learning from Other Disciplines, including Medicine and

Journalism

10:00 Comparisons to Other Fields Part 1 Chair: Stephen Marrin (Brunel University)

David Chuter (Center for Security Sector Management; Cranfield University): Intelligence Analysis

as a Type of Information Processing

Michael Robinson (Deloitte and Touche, LLP): Intelligence Analysis and Digital Forensics

Carl Jensen (University of Mississippi): Intelligence Analysis and Futures Methodology

11:00 Break

11:30 Comparisons to Other Fields Part 2 Chair: Carl Jensen (University of Mississippi)

Anne Bishop (Snell & Wilmer, LLP, Phoenix AZ): Intelligence Analysis and Legal Reasoning

Anthony C. Cain (Air Force Research Institute): Intelligence Analysis and Professional Journal

Publishing

William Lawhead (University of Mississippi): Seeing Is Not Believing: Insights for Intelligence

Analysis from Professional Magicians.

12:30 Lunch

1:30 Intelligence Analysis and Social Science

Chair: Stephen Marrin (Brunel University)

Stefania Paladini (Coventry University): Bayesian analysis in intelligence and social sciences: a tool

for all trades?"

2:30 Break

3:00 Intelligence Analysis and History Chair: Philip Davies (Brunel University)

Melissa Graves (University of Mississippi): Intelligence Analysis and Historiography

Howard Clarke (The Intelligence Solutions Group) Seekers after Truth: Cross-Disciplinary Insights

for the Intelligence Profession from the Biblical and Theological Studies Discipline

WJR Gardner (Naval Historical Branch, UK Ministry of Defence): The Unreliable Memoirs of an

Applied Historian … and Operational Analyst and …

4:00 Break

4:15 Intelligence Analysis and Perception Chair: Melissa Graves (University of Mississippi)

Patricia Rooney (National Ground Intelligence Center, US Army) and Peter Ronayne (Federal

Executive Institute): Neuroscience in Intelligence Analysis: Understanding How the Brain Works and

Its Impact on Analytic Decision Making

Musa Tuzuner (Intelligence Studies Research Center; Turkish National Police Academy): Multi-

Level Cognitive Biases in the Chain of Intelligence Flow—From Source to Consumers

Mandeep Dhami (University of Surrey): Intelligence Analysis and Decision Science (Putting the

Science into the Art of Intelligence Analysis)

5:30 Day One Adjourns

3

Conference agenda Day 2: Friday 13 July

8:00 Registration and Coffee

9:30 Welcome Part 2

9:45 Evaluating Intelligence John Kringen (former CIA Director of Intelligence; Institute for Defence Analysis): Judging

Intelligence Success and Failure

10:45 Evaluating Intelligence

Chair: Philip Davies (Brunel University)

Stephen Marrin (Brunel University): Evaluating the Quality of Intelligence Analysis: By

What (Mis) Measure?

Rhiannon Gainor and France Bouthillier (McGill University): Measuring Intelligence

Success: Conceptual and Methodological Challenges of Intelligence Measurement

11:15 Break

11:30 Intelligence Analysis and the Dismal Science Chair: Julian Richards (University of Buckingham)

Michael Herman (Oxford University): Intelligence Analysis and Government Statistics

Nick Hare (UK Ministry of Defence): Intelligence Analysis and Economics

12:30 Lunch

1:30 Improving Communication Chair: Julian Richards (University of Buckingham)

Sean Newman (Fire Department of New York): How Adherence to Journalistic Standards

Positioned FDNY Watchline at the Forefront of Fire Service Analytical Intelligence

Douglas Bernhardt (South Africa: Wits Business School): Intelligence Reporting: What

Works, What Doesn't and How to Fix It

Gabriel Sebe (Bucharest University): Political Marketing Intelligence

2:30 Break

3:00 Applying Knowledge from Other Fields to Increase Understanding Peter Nardulli and Dan Roth (University of Illinois): Advancing our Understanding of Global

Patterns of Civil Unrest: An Interdisciplinary Effort

Kathleen Vogel (Cornell University): Current Intelligence Reporting and the H5N1 Bird Flu

Virus: Insights from the Field of Science and Technology Studies

4:00 Break

4:15 Practitioner views, summary, and next steps TBD

5:30pm Conference Adjourns

4

ABOUT THE SPEAKERS Douglas Bernhardt lectures in the subject of Competitive Intelligence at Wits Business School,

Johannesburg, the University of Stellenbosch Business School, and the Nelson Mandela Metropolitan

University Business School, Port Elizabeth. He is also an adjunct instructor in Competitive Intelligence

for Mercyhurst University. Previously, Douglas served as an adjunct professor of Business Intelligence &

Corporate Security at the European campus of the Thunderbird School of Global Management. He has

also taught at IMD, Lausanne and the Rotterdam School of Management. He is the author of three books,

including Competitive Intelligence: Acquiring and using corporate intelligence and counterintelligence,

published in 2003 by Financial Times Prentice-Hall. From 1993-2001 Douglas served as Managing

Director for one of Europe’s leading competitive intelligence consultancies, Business Research Group

SA, in Geneva and London, supporting the CI requirements for major firms in industries such as

pharmaceuticals, telecommunications, and FMCGs. He was a member of the Board of Directors of the

US-based Society of Competitive Intelligence Professionals (SCIP) from 1996-1999. His early

commercial experience includes the defence industry and foreign trade.

Anne W. Bishop is an attorney at Snell & Wilmer, LLP, in Phoenix, Arizona. Prior to attending law

school, Ms. Bishop worked at the National Security Agency at Ft. Meade, Maryland where she held

positions as a linguist and analyst, and taught intelligence analysis to other analysts. Following a

fellowship working for a New York Congresswoman, she worked in NSA’s legislative affairs office,

working with the House Permanent Select Committee on Intelligence Issues. Ms. Bishop’s legal practice

focuses on health care compliance, health care litigation, and business disputes. She advises physicians

and hospitals on various health-care related issues. Ms. Bishop is also a member of the privacy and data

security group. Ms. Bishop is a frequent writer and speaker on health care compliance and legal ethics.

Ms. Bishop graduated in 1986 from the University of Pennsylvania with a degree in international

relations & in 2006 from the Sandra Day O’Connor College of Law at Arizona State University.

Dr. Anthony C. (Chris) Cain is Chief of Academic Affairs at The Air University, Maxwell Air Force

Base, Alabama. He received his M.A. and Ph.D. in Military History from The Ohio State University, and

also holds Masters Degrees from the Air War College and Embry-Riddle Aeronautical University, as well

as a B.S. from Georgia State University. Dr. Cain is a graduate of the Air Command and Staff College

(ACSC), and the Air War College. Prior to assuming his present duties, he served as the Deputy Director

of the Air Force Research Institute (AFRI), Dean of AFRI, Chief of AFRI’s Outreach Division, founding

Editor-in-Chief of Strategic Studies Quarterly, Research Director and Dean of Education & Curriculum at

ACSC, and as the Chief, Professional Journals Division & editor of the Air and Space Power Journal at

the College of Aerospace Doctrine, Research and Education. Dr. Cain is the author of The Forgotten Air

Force: The French Air Force and Air Doctrine in the 1930s (Smithsonian Institute Press, 2002), as well

as book chapters and articles on issues dealing with airpower, professional military education, effects-

based operations, regional threats from weapons of mass destruction, & military history.

Dr. David Chuter worked for more than thirty years for the UK government in the defence and security

area. He was involved, among other subjects, in the negotiations leading to the Maastricht Political Union

Treaty, and subsequent discussions about a European security and defence policy, as well as arms control,

weapons of mass destruction, and war crimes and transitional justice. His last job was as Special Adviser

to the Policy Director of the French Ministry of Defence in Paris. He took early retirement at the end of

2008 and is now an author, lecturer, translator and consultant based in Paris. Dr Chuter is a lecturer at

Sciences Po in Paris, where he teaches courses in Intelligence and Policy-making, and in Crisis

Management. He is also Senior Research Fellow at the Centre for Security Sector Management at

Cranfield University, on whose behalf he teaches security-related subjects in various countries, and Chair

of the Governing Board of the International Security Information Service Europe. He is the author of four

books, and many articles, most recently Governing and Managing the Defence Sector (2011). His current

research interest is the politics and practical problems of the Rule of Law in the Security Sector.

5

Howard Clarke, MCS, CCA is an internationally experienced law enforcement intelligence practitioner.

During his thirty-year law enforcement career Howard held senior intelligence analysis positions with law

enforcement agencies in Australia and Canada. For the past decade he has specialized in strategic

intelligence analysis applications in law enforcement and homeland security and he remains active in

intelligence analysis training and consulting work. He has taught extensively on strategic intelligence

analysis and open source intelligence issues in the U.K., Europe, Asia and North America and is a

published author on Insider Threat issues. He holds a Master of Christian Studies from Regent College

International Graduate Theological School. Howard is currently an intelligence analysis instructor at the

British Columbia Institute of Technology and the Justice Institute of British Columbia in Canada and he

formerly held adjunct faculty status with Henley-Putnam University in the U.S. He is a Lifetime Certified

Criminal Analyst, a former member of the Board of Governors of the Society of Certified Criminal

Analysts (SCCA), a member of the International Association of Law Enforcement Intelligence Analysts

(IALEIA) and the International Association for Intelligence Education (IAFIE).

Dr. Philip H.J. Davies is a political sociologist specialising in the study of national intelligence

institutions. He is Director of the Brunel Centre for Intelligence and Security Studies (BCISS). In 2010

BCISS was awarded University Interdisciplinary Research Centre status and now includes scholars from

across the University from departments and schools as diverse as Engineering and Design, Economics

and Law in addition to its core team based in Politics and History. During 2009-10, Dr. Davies headed

BCISS work on the new UK military Joint Intelligence Doctrine. On this project, Brunel was selected by

the Ministry of Defence as a full partner with its Development, Concepts and Doctrine Centre and

Defence Intelligence (former the Defence Intelligence Staff). This work has involved both the

fundamental rethinking and reformulation of intelligence doctrine (JDP 2-00 Understanding and

Intelligence in the Joint Operating Environment) but also the articulation of an entirely new doctrine on

‘understanding’ (JDP 04 Understanding). Before that Dr. Davies ran an ESRC-funded international

seminar series entitled ‘Intelligence and Government in the 21st Century’ working in collaboration with

Dr. Robert Dover at Loughborough. He has also recently completed a major and intensive comparative

study of British and American national intelligence has been published by Praeger Security International

in 2012. In 2004 Dr. Davies designed and, until 2010, convened Brunel’s highly successful MA in

Intelligence and Security Studies, including its ground-breaking Brunel Analytical Simulation Exercise.

BASE is term-long practical in strategic intelligence assessment in which students are divided into

drafting teams modelled on the UK Joint Intelligence Organisation analytic staff and produce JIO-style

assessments on live, real-world topics using open sources.

Dr. Mandeep K. Dhami is a Reader in Forensic Psychology at the University of Surrey. Previously she

held academic appointments at the University of Cambridge (UK), University of Victoria (Canada), the

University of Maryland (USA), and the Max Planck Institute for Human Development (Germany). She

has worked as a Senior Scientist for the Defence Science and Technology Laboratory (MoD). Dr. Dhami

has a PhD in Psychology and an MA in Criminology. Her research draws conceptual connections between

Psychology (esp. decision science and social cognition) and Criminology. She has examined topics such

as decision making, risk, and forecasting. She has applied these issues to the criminal justice system, and

more recently to the defence and security sectors (with a focus on intelligence analysis and cyber issues).

To-date, she has over 65 publications, and is lead editor of a book entitled “Judgment and Decision

Making as a Skill: Learning, Development and Evolution” published by Cambridge University Press in

2011. Her research has won several awards including from the European Association of Decision Making

and the Society for the Psychological Study of Social Issue (SPSSI, Division 9 of the American

Psychological Society). Dr. Dhami is Fellow of SPSSI (Division 9 of the American Psychological

Association), and is on the editorial board of several journals including Perspectives on Psychological

Science. She has provided advice and consultancy to various government bodies including the Home

Office, Ministry of Justice, Government Communications Headquarters, and the Metropolitan Police

(New Scotland Yard).

6

Rhiannon Gainor is a PhD candidate at McGill University’s School of Information Studies, a

McConnell Foundation Fellow in 2010/2011, and a recipient of Québec’s Fonds de recherche Société et

culture (FQRSC) grant. Her research interests are knowledge management, competitive intelligence, and

information metrics. She has a Master’s of Library and Information Studies, and Master’s of Arts in

Humanities Computing from the University of Alberta.

W J R (Jock) Gardner served in the Royal Navy for 30 years, specialising in anti-submarine warfare.

He served on operational, analysis and intelligence staffs. In 1989 he was awarded an M Phil in

International Relations at the University of Cambridge, writing a dissertation on Soviet nuclear ballistic

missile submarines. He was also the Editor of The Cambridge Review of International Affairs. Leaving

the Navy in 1994, he joined the Naval Historical Branch as a Historian and taking an especial interest in

World War II, antisubmarine warfare and intelligence. His publications include Anti-submarine Warfare

(Brassey’s, 1996) and Decoding History: the Battle of the Atlantic and Ultra (Naval Institute Press,

1999). He has lectured on naval historical and intelligence topics from Moscow to Monterey. He is the

external examiner at Brunel University for the MA in Intelligence and Security Studies, a member of the

Editorial Board of The Mariner’s Mirror and a Fellow of the Royal Historical Society.

Melissa Graves, JD, MA serves as Project Coordinator and Instructor at the University of Mississippi’s

Center for Intelligence and Security Studies. Among her accomplishments, she and her co-developer

Walter Flaschka have designed and implemented the Days of Intrigue, a realistic practical exercise that is

conducted yearly at UM and which involves numerous intelligence community agencies. Ms. Graves

received her BA with a double major in English and Communications from Hardin-Simmons College, an

MA in History from UM, and a JD from the UM School of Law. She is presently pursuing a Ph.D. in

History. Her work will appear in upcoming publications, including Introduction to Intelligence Studies

(Carl Jensen, David McElreath, and Melissa Graves) and Intelligence Studies in Britain and the US:

Historiography since 1945 (ed. Christopher Moran).

Nick Hare is a professional economist, and has worked in a number of roles in the UK Ministry of

Defence (MOD) since graduating in 1999 with an MA in Philosophy and Economics from the University

of Edinburgh. He is currently head of the Futures and Analytical Methods (FAM) team in the MOD. The

FAM team works to promote the aims of the Professional Head of Defence Intelligence Analysis. This

involves encouraging analysts to use structured and auditable methods to refine intelligence requirements,

identify assumptions, generate hypotheses and scenarios, test hypotheses using data, collect information,

and describe conclusions using appropriate probabilistic language. The FAM team also researches new

and developing tools and approaches that are of potential use to intelligence analysts, and works with

other departments to promote professionalisation of analysis across government.

Michael Herman served from 1952 to 1987 in Britain's Government Communications Headquarters,

with secondments to the Cabinet Office (as Secretary of the Joint Intelligence Committee) and to the

Ministry of Defence.. His Intelligence Power in Peace and War was published in 1996 and has been

regularly reprinted. His Intelligence Services in the Information Age was published in 2001, and he has

co-edited and contributed to Intelligence in the Cold War: What Difference Did It Make (in publication).

He is currently an Honorary Departmental Fellow at Aberystwyth University and an Associate Member of

Nuffield College Oxford, as Founder Director of the Oxford Intelligence Group. He is an Honorary D.Litt

of Nottingham University.

7

Dr. Carl J. Jensen III. is a 1978 graduate of the U. S. Naval Academy. He served in the Navy from 1978

until 1983, first aboard the nuclear fleet ballistic missile submarine USS George Washington Carver and

then as an aide to the Commander of Submarine Group Five. Dr. Jensen graduated from FBI New Agents

Training in 1984 and served as a field agent in Atlanta, Georgia, Monterey, California, and Youngstown,

Ohio. In August 1992, Dr. Jensen reported to the FBI Laboratory where he received certification as a

Racketeering Records Examiner. In June 1997, Dr. Jensen reported to the Behavioral Science Unit at the

FBI Academy in Quantico, Virginia, where he instructed senior police officials at the FBI National

Academy, conducted research, provided consultation, and served as Assistant Unit Chief. Upon his

retirement from the FBI in 2006, Dr. Jensen joined the RAND Corporation as a Senior Behavioral

Scientist. In 2007, he joined the Legal Studies faculty at the University of Mississippi, where he currently

serves as director of the Center for Intelligence and Security Studies. He is the 2012 recipient of the

Outstanding Instructor of the Year Award from the International Association for Intelligence Education, a

2008 recipient of the Thomas Crowe Outstanding Faculty Award from the University of Mississippi and a

2004 recipient of the Jefferson Award for Outstanding Research from the University of Virginia. Dr.

Jensen holds a Master of Arts degree from Kent State University and a Doctor of Philosophy degree from

the University of Maryland. He has instructed throughout the world and is the author of numerous

articles, books, book chapters and technical reports.

Dr. John A. Kringen is a Research Staff Member with the Institute for Defense Analyses in Alexandria

Virginia. He recently retired from the Central Intelligence Agency after more than three decades of

government service. During 2005-2008 he served as the Director of Intelligence at CIA where he

managed and led the activities of several thousand CIA analysts and staff. Prior to becoming Director of

Intelligence, he served in a variety of managerial assignments in that Directorate, including as Director of

the CIA’s Crime and Narcotics Center. Beyond CIA, he served twice overseas in Europe, most recently

as a senior intelligence advisor to United States European Command from July 2008 to August 2011. In

the late 1990s, he was head of imagery analysis at the predecessor organization to the National Geospatial

Intelligence Agency. He received a Ph.D. from the University of Minnesota in 1978.

Dr. William Lawhead retired from the University of Mississippi in summer 2012 after having completed

seven years as Chair of the Department of Philosophy and Religion. He also completed forty-one years of

teaching philosophy, thirty-two of them at UM. Among the courses he has taught are history of

philosophy, logic and critical thinking, and philosophy of science. He has served on the advisory board

of the University of Mississippi’s Center for Intelligence and Security Studies and has participated in and

given papers at 5-Eyes Conferences. He is a member of the International Association for Intelligence

Education. He has published two books. These are a history of Western philosophy, The Voyage of

Discovery, 3rd

ed. (Wadsworth, 2007) and an introduction to philosophy, The Philosophical Journey, 5th

ed. (McGraw-Hill, 2011). He is working on a third book, Taking Philosophy Seriously, for Oxford

University Press. Dr. Lawhead received his Ph.D. in philosophy from the University of Texas, Austin.

Dr. Stephen Marrin is a Lecturer in the Department of Politics and History at Brunel University in

London, England. He received his PhD and MA in foreign affairs from the Department of Politics at the

University of Virginia. He has written about many different aspects of intelligence analysis, including

new analyst training at CIA’s Sherman Kent School, the similarities and differences between intelligence

analysis and medical diagnosis as a source of ideas for improving the quality of future intelligence

analysis, and the professionalization of intelligence analysis. In 2004 the National Journal profiled him as

one of the ten leading experts on intelligence reform. He previously worked as an analyst at both the US

Central Intelligence Agency and the US Government Accountability Office, and taught in the Intelligence

Studies Department at Mercyhurst University.

8

Dr. Peter F. Nardulli is Professor of Political Science and Law at the University of Illinois at Urbana-

Champaign, the founding Director of the Cline Center for Democracy, and the editor of a book series with

the University of Illinois Press: Democracy, Free Enterprise and the Rule of Law. He has been on the

faculty at UIUC since 1974 and served as department head in Political Science from 1992 until 2006.

Nardulli is the author of six books on various aspects of the legal process and empirical democratic

theory. He has authored a number of articles in journals such at the American Political Science Review,

Public Choice, Political Communication, Political Behavior and a number of law reviews. Nardulli is

currently directing a global study, the Societal Infrastructures and Development Project (SID). SID uses a

number of technologically advanced, innovative methodologies to examine the impact of political, legal

and economic institutions on a wide range of societal development indicators (economic growth, human

rights, societal stability, environmental quality, educational attainment etc.). Current projects involve

using data from the SID project’s Social, Political and Economic Event Database (SPEED) project to

examine the impact on civil unrest of such things as climate change, natural resources, socio-cultural

animosities and political institutions.

Captain Sean S. Newman, a 16-year veteran of the Fire Department City of New York (FDNY), has

worked most of his career in Manhattan, and is a charter member of FDNY Center for Terrorism and

Disaster Preparedness (CTDP), opened in 2004. Captain Newman has been editor and contributing writer

for FDNY Watchline since the intelligence product’s inception in 2008. Released in December 2011, he

co-wrote the “FDNY Counterterrorism and Risk Management Strategy.” Captain Newman graduated with

honors from the Center for Homeland Defense and Security (CHDS) at the U.S. Naval Postgraduate

School with a master’s degree in security studies. Aside from his professional fire experience, Captain

Newman is a former journalist on the Reuters equities' desk in NYC.

Dr. Stefania Paladini is Associate Head of Department in Strategy and Management at Coventry

University. Before joining academia she has spent 7 years appointed as trade commissioner in East Asia,

where she has also obtained a PhD from City University of Hong Kong in security studies. An economist

and a statistician by professional practice, she has also published extensively on environmental and

economic security and given a number of policy briefings in Italy at ministerial level. She has a long term

interest in intelligence studies, where she is now working on the application of Bayesian analysis and

other quantitative methods.

Dr. Julian Richards successfully completed a doctorate in political violence in Pakistan, at Cambridge

University, in 1993. He then spent nearly 20 years working in intelligence and security for the British

government. In 2008, he co-founded the Centre for Security and Intelligence Studies (BUCSIS) at the

University of Buckingham, where he teaches on two MA programmes in Intelligence Studies and Global

Security, and supervises a number of PhD students. His affiliations include Associate of the Pakistan

Security Research Unit in Bradford University, a member of the European Ideas Network, and the

editorial board of the Spanish journal Intelligencia y seguridad. Julian Richards is the publisher of two

books: The Art and Science of Intelligence Analysis (Oxford University Press, 2010); and A Guide to

National Security: Threats, Responses and Strategies (Oxford University Press, 2012); in addition to a

number of papers on a range of security and intelligence issues. He is also a regular media commentator

on security and intelligence issues for BBC television and radio, in addition to appearances on Al Jazeera

and the Islam Channel. His current research interests include attitudes towards surveillance in

contemporary society; the rise of cyber-threats and cyber-security; security issues in Pakistan and

Afghanistan; and violent extremism in Europe.

9

Michael Robinson is a Specialist Leader with Deloitte and Touche, LLP, where he conducts computer

and mobile device forensics. Prior to joining Deloitte and Touche, LLP, Robinson conducted forensic

examinations for the Federal Bureau of Investigation. He also previously worked as the Chief Information

Officer of the U.S. Department of Defense's Business Transformation Agency, where he was responsible

for all Information Technology and Information Assurance activities, as well as forensic examinations.

Robinson is an adjunct professor at Stevenson University in the School of Graduate and Professional

Studies, where is the program coordinator of the Cyber Forensics degree. He is also an adjunct professor

at George Mason University, where he teaches mobile device forensics.

Dr. Peter V. Ronayne is a senior faculty member at The Federal Executive Institute (FEI), an executive

leadership education center within the U.S. Office of Personnel Management. Pete formerly directed

FEI’s flagship Leadership for a Democratic Society program, co-founded FEI’s Center for Global

Leadership and FEI’s Leadership Horizons Series for the Senior Executive Service. He joined FEI after

studying, teaching, and researching at the University of Virginia. Pete is a leading voice in the public

sector on generational and demographic issues and their impact on leadership and organizations. He also

writes, researches, and speaks widely on issues of global leadership, neuroscience and leadership, and the

future of public service and governance. A former Presidential Management Fellow, Pete is an adjunct

professor at the University of Virginia where he teaches undergraduate courses in world politics,

diplomatic history, and leadership. Pete is currently at work on a biography of Secretary of State

Madeleine Albright to be published in 2013. He is also the author of Never Again?The United States and

the Prevention and Punishment of Genocide since the Holocaust, the first book to chronicle America’s

complicated and conflicted response to the crime of genocide since 1945. Pete earned a PhD in

International Affairs at the University of Virginia and a BS in Languages and Linguistics at Georgetown

University. Prior to his executive education career, Pete worked in the fields of international emergency

medical assistance and international development, where he worked on small-scale, sustainable and

replicable technologies for developing nations.

Dr. Patricia J. Rooney is currently the Director of Training at the U.S. Army’s National Ground

Intelligence Center (NGIC). Prior to joining NGIC, PJ served as senior faculty at the Federal Executive

Institute where she taught executive leadership courses with a focus on neuroscience. Before Joining FEI,

PJ served in a number of leadership positions within the U.S. Department of Defense (DoD). While at the

National Geospatial-Intelligence Agency (NGA), PJ was the Deputy Director of the field operations

office co-located with NGIC. She led the National Geospatial-Intelligence College as Deputy Director,

and has served as the Dean for NGA’s School of Leadership and Professional Studies. In addition to

experience in the Federal government, PJ served as a Senior Associate with SENSA Solutions, working

with government leaders on strategic planning, business process re-engineering, leadership change

strategies, and senior executive coaching. Before joining the intelligence community, PJ served on the

faculty at the United States Military Academy teaching in the Leadership and Behavioral Science

Department; and served as a counselor for Victims of Violent Crime with the Metropolitan Nashville

Police Department. PJ earned a PhD in Education from the University of Virginia, an MS in Strategic

Intelligence from the Joint Military Intelligence College, a MEd from Vanderbilt University, and a BA in

Political Science from the University of North Carolina at Chapel Hill.

10

Dr. Sorin-Gabriel Sebe teaches in the Department of Political Science at the University of

Bucharest in Romania. He graduated from the Faculty of Mathematics, University of Bucharest in 1986

(the graduation thesis: „Non-Riemannian Geometries in Dislocation Theory”). From 1990 to 1995 he

worked as a Research Associate at GPM2 Laboratory in Grenoble, France, in the realm of mathematical

modeling of irreversible phenomena. In 1996 he joined the Faculty of Political Science, University of

Bucharest (FSPUB) and currently, beyond teaching, serves as Vice-Dean. He received his PhD in

sociology in 2004 (the final thesis: „Subjective Quality of Life Modeling”) from the Faculty of Sociology,

University of Bucharest. He was a general manager at INSOMAR (2004-2006). His research as a

political science academic has primarily been concerned with the processes associated to the

democratization of former non-democratic regimes, such as the reform of the intelligence apparatus and

the development of a political market. He favors a trans-disciplinary approach on research, having as

current interest the development of a curriculum for the study of intelligence in academic settings, in a

political science faculty, in a former communist country.

Dr. Musa Tuzuner had a long and varied experience in the intelligence field in Turkey. PhD in Political

Science (Kent State University, 2009 with Best Dissertation Award); Assistant Professor, Turkish

National Police Academy (2009–); Founding Director, Intelligence Studies Research Center (2009–) &

Intelligence Studies Master Program (2010–), TNP Academy; current main interests: national and

international security behavior, theorizing Turkish foreign policy, and international event data

developments. Most recent book: Intelligence Cooperation Practices in the 21st Century: Towards a

Culture of Sharing (IOS Press, 2010). Most recent article: Quantifying intelligence cooperation: The

United States International Intelligence Behavior (USIIB) dataset, with Ersel Aydinli (Journal of Peace

Research 48(5) 673–682, 2011).

Dr. Kathleen Vogel is an associate professor at Cornell, with a joint appointment in the Department of

Science and Technology Studies and the Judith Reppy Institute for Peace and Conflict Studies. Vogel

holds a Ph.D. in biological chemistry from Princeton University. Prior to joining the Cornell faculty,

Vogel was appointed as a William C. Foster Fellow in the U.S. Department of State’s Office of

Proliferation Threat Reduction in the Bureau of Nonproliferation. Vogel has also spent time as a visiting

scholar at the Cooperative Monitoring Center, Sandia National Laboratories and the Center for Nonprolif-

eration Studies, Monterey Institute of International Studies.

11

Understanding and Improving Intelligence Analysis:

Learning from other Disciplines

12-13 July, 2012 * London, England * RAF Club

RELATED READINGS

Benson, Sumner. "The Historian as Foreign Policy Analyst." The Public Historian 3, no. 1 (1981): 15-25.

(HISTORY/HISTORIOGRAPHY)

Converse, Ray & Randy Pherson. Intelligence and Medicine: Parallel Cognitive Traps. Unpublished

paper. Pherson Associates. 2009. 1-6. (MEDICINE/PSYCHOLOGY/COGNITION)

Fisher, Rebecca and Rob Johnston. Is Intelligence Analysis a Discipline? Roger George and James Bruce

(Eds). Analyzing Intelligence. Georgetown University Press. Washington DC. 2008. 55-68.

(LAW/LEGAL PROFESSION/LIBRARY SCIENCE/MEDICINE)

Heuer, Richards J. “Adapting Academic Methods and Models to Governmental Needs.” Quantitative

Approaches to Political Intelligence: The CIA Experience. (Ed. Richards J. Heuer). Westview Press.

1978. (1-10). (SOCIAL SCIENCE METHODS)

Johnston, Rob. Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study. Washingon,

DC: Central Intelligence Agency, Center for the Study of Intelligence, 2005. (ANTHROPOLOGY)

Kerbel, Josh. “Lost for Words: The Intelligence Community’s Struggle to Find its Voice.” Parameters.

2008. 102-112. (MEDICINE)

Knorr, Klaus. “Foreign Intelligence and the Social Sciences.” Research Monograph No. 17. Center of

International Studies. Princeton University. June 1, 1964. (SOCIAL SCIENCE METHODS)

Laqueur, Walter. "The Question of Judgment: Intelligence and Medicine." Journal of Contemporary

History 18 (Oct. 1983): 533-548. (MEDICINE)

Leslau, Ohad. "Intelligence and Economics: Two Disciplines with a Common Dilemma."

International Journal of Intelligence and Counterintelligence 20, no. 1 (Spring 2007): 106-

121.(ECONOMICS/ECONOMIC ADVISORS)

12

Marrin, Stephen, & Jonathan Clemente. "Improving Intelligence Analysis by Looking to the Medical

Profession." International Journal of Intelligence & Counterintelligence 18/4 (Winter 2005-06): 707-729.

(MEDICINE)

Marrin, Stephen, and Jonathan Clemente. "Modeling an Intelligence Analysis Profession on Medicine."

International Journal of Intelligence and Counterintelligence 19/4 (Winter 2006-2007): 642-665.

(MEDICINE)

Marrin, Stephen. “Intelligence Analysis: Structured Methods or Intuition?” American Intelligence

Journal. Vol. 25.No. 1.(Summer 2007).7-16. (MEDICINE)

Nolte, William. "Thinking about Rethinking: Examples of Reform in Other Professions." Studies in

Intelligence 52, no. 2 (Jun. 2008): 19-25. (GENERAL)

Pritchard, Matthew C. & Michael S. Goodman (2008): Intelligence: The Loss of

Innocence, International Journal of Intelligence and CounterIntelligence, 22/1, 147-164.

(ARCHAEOLOGY)

Rieber, Steven, and Neil Thomason. "Toward Improving Intelligence Analysis: Creation of a National

Institute for Analytic Methods." Studies in Intelligence 49, no. 4 (2005): 71-77. (MEDICINE)

Rodgers, R. Scott. "Improving Analysis: Dealing with Information Processing Errors." International

Journal of Intelligence and Counterintelligence 19/4 (Winter 2006-2007): 622-641. (PSYCHOLOGY)

Spivey, Robin V. "The Devil Is in the Details: The Legal Profession as a Model for Authentic Dissent."

International Journal of Intelligence and Counterintelligence 22/4 (Winter 2009): 632-651.

(LAW/LEGAL REASONING)

Weiss, Charles. "Communicating Uncertainty in Intelligence and Other Professions." International

Journal of Intelligence and Counterintelligence 21, no 1 (Spring 2008): 57-85.

(GENERAL/SCIENCE/LAW)

Young, David. Scholarship, Intelligence, and Journalism. Reuters Fellows Lecture. March 2004.

(JOURNALISM)

13

Paper and Presentation Abstracts Understanding and Improving Intelligence Analysis:

Learning from other Disciplines

12-13 July, 2012 * London, England * RAF Club

Douglas Bernhardt (South Africa: Wits Business School): Intelligence Reporting: What

Works, What Doesn’t and How to Fix It: This paper argues that however whatever the quality of the

findings and analysis of an intelligence deliverable, it will often fail to achieve its main objective; that is, to provide

a decision-maker with product that he or she finds both compelling and relevant to his or her specific decision

challenges. This occurs mainly for two reasons: (1) the ‘disconnect’ which often exists between intelligence

consumers and producers and (2) the ‘packaging’ of finished intelligence as a set of dry, colourless news and facts.

This author, a long-time practitioner and lecturer in the field of Competitive Intelligence, has previously described

the phenomenon of the sub-optimal relationships which exist in many firms between Competitive Intelligence units

and their executive customers. In this paper he not only revisits and updates earlier discussions, but adds a fresh

dimension; one which centres around the process of persuasion. In short, if an intelligence product is not persuasive

(think the PDB of 6 August 2001), it has no value; it’s just another report to digest. This paper will consider the

factors which can and should make an intelligence briefing, or report, persuasive; borrowing, in part, from the fields

of advertising and sales. It will also focus on the unique contribution that human source information (HUMINT)

makes to intelligence analysis and reporting OSINT is important, but it’s not enough. The paper/presentation we

propose will explore two principle topics: (1) what are the issues. involved in, and what needs to be done to

overcome, the challenges inherent in the producer-consumer relationship? and (2) what tools can we apply to

improve, and sometimes totally recast, the packaging and delivery of intelligence product that makes a difference,

that serves as a springboard for its users to gain ‘competitive advantage’? Although this presentation will be

discussed within a corporate sector context, the key lessons are equally relevant to the domains of national security

and law enforcement.

Anne Bishop (Snell & Wilmer, LLP, Phoenix AZ): Intelligence Analysis and Legal

Reasoning: This presentation will explore the similarities between legal reasoning and intelligence analysis. The

presenter is a practicing attorney who spent about 15 years in the U.S. intelligence community, working as an

analyst and linguist and training intelligence analysts. Ms. Bishop will focus on the similarities in acquiring relevant

information, evaluating sources, and deriving meaning from that information. In both intelligence analysis and legal

research, locating the relevant information is critically important. Ms. Bishop will discuss access, search strategies,

and ways to validate search results. In addition, Ms. Bishop will discuss the relatively static hierarchy of sources

that exists in the legal research realm. However, the bulk of the presentation will focus on analytic techniques that

are central to legal analysis-- analogies, extrapolation, legal reasoning, and the importance of context. Lawyers are

taught in law school to draw analogies between their current cases and existing case law precedent. By creating a

compelling analogy, a lawyer is likely to prevail in a court. Similarly, by drawing analogies to prior events, an

intelligence analyst can analogize what is transpiring today based on similar prior events. In addition, Ms. Bishop

will discuss extrapolation and its value to both intelligence analysis and legal reasoning. That is, in legal reasoning,

there is often no legal principal that is completely similar. Lawyers learn to find a case that is analytically similar

and extrapolate the principles to the situation at hand. Similarly, intelligence analysts can look at an existing set of

facts and extrapolate from known facts into the unknown. In legal reasoning, lawyers look at two existing cases and

draw inferences regarding how those two cases fit together in order to determine how a court should rue on a current

case. This technique could be used predicatively by intelligence analysts. Finally, Ms. Bishop will discuss the

importance of context. Context is key in determining a legal outcome. The same is true for intelligence analysis.

Determining the status quo is critical in both context in order to figure out the potential outcomes.

14

Anthony C. Cain (Air Force Research Institute): Intelligence Analysis and Professional

Journal Publishing: Publishing professional military journals requires many of the same processes and

decision capabilities found in intelligence analysis. Professional journals, by their nature, can narrowly

focus on technical or tactical subjects, but the selection and editing of content requires broad knowledge

in several areas. For example, journals that focus on operational issues may receive article submissions

that deal with the full spectrum of service or joint capabilities. Journal editors will rarely be trained or

equipped to evaluate the credibility and merits of such a broad spectrum of content. Therefore, creating

mechanisms to ensure the credibility and quality of journal content form the essential tasks for

professional journal editing. Those same mechanisms could be adapted to serve intelligence analysis

processes. Specific cases from the author’s experience as the editor of Air and Space Power Journal and

as founding editor-in-chief of Strategic Studies Quarterly will illustrate the challenge and solutions

adopted to preserve the professional credibility of both journals.

David Chuter (Center for Security Sector Management; Cranfield University): Intelligence

Analysis as a Type of Information Processing Although intelligence analysis is obviously a special case of the collection, processing and use of

information, it is not entirely unique. Indeed, we carry out analogous tasks to intelligence analysis all the

time, not only professionally, but in our personal lives as well. Intelligence analysis can be seen as a type

of information processing which has a special combination of characteristics, generally not found together

elsewhere. We can analyse all information processing under three types of heading. The first is the degree

of applicability, which is to say how far the process is designed to lead to an operational outcome. At one

extreme, practitioners of emergency medicine have a short period in which to gather, process and act on

information in order to save a life. At the other extreme, scientists, archaeologists or sociologists may

spend years on a research programme which simply validates prior assumptions, and does not have any

operational output at all. Intelligence is in principle only collected in response to specific tasking to

answer specific questions, and so in principle is a high-applicability process. The second is the degree of

completeness of the information which can reasonably be desired. Surgeons contemplating major, non-

acute surgery, will exhaustively research everything they can, to make sure that what they do is as

effective as possible. At the other extreme, many decisions are successfully taken in everyday life on the

basis of very incomplete information: if you arrive at an airport and realize that you have left your adaptor

plug at home, you will decide to buy the first thing you see that functions correctly. There are many

intermediate cases: historical research, for example, can never in principle be complete or definitive, no

matter how long it continues. Criminal trials generally only consider enough evidence to establish guilt

beyond a reasonable doubt, and some potential crimes may not even be charged. Intelligence is by

definition always incomplete, and its reliability is always suspect. The third is the need for and

expectation of accuracy. Most medical decisions have to be very accurate, since otherwise the patient will

be harmed, and there are professional sanctions for those who make mistakes. At the other extreme,

journalism and the productions of advocacy NGOs are seldom expected to be fully accurate, and there are

few professional sanctions for inaccuracy or even downright untruths. There are intermediate cases such

as investment advice, where there is an expectation of at least broad accuracy, but few sanctions in its

absence. There are also cases (like the conclusions of juries in criminal trials) where the degree of

accuracy is not knowable, and all one can ask is whether the verdict is reasonable in the circumstances.

Intelligence is generally an area with high (and often excessive) expectations of accuracy, and a great

requirement for it. The special combination of high applicability, lack of completeness and significant

demand for accuracy makes intelligence analysis a special and interesting case.

15

Howard Clarke (The Intelligence Solutions Group) Seekers after Truth: Cross-Disciplinary

Insights for the Intelligence Profession from the Biblical and Theological Studies Discipline Biblical hermeneutics, or exegesis, is an interpretative activity that contains some significant parallels

with important facets of the intelligence analysis endeavor. The term exegesis refers to careful

investigation of the original meaning of texts in their historical and literary contexts. The process involves

asking analytical questions about various aspects of the source materials and their contexts, and it implies

careful, detailed analysis. As an analytic discipline, effective biblical interpretation requires a capacity to

deal with challenging data sets, complex source evaluation circumstances, and it seeks to provide insights

which will shape and influence the beliefs and actions of a significant community (i.e. the consumers). In

principle, a process not dissimilar to the Intelligence Cycle is employed and at various times and in

various contexts there is a need to utilize techniques which can be construed as forms of HUMINT,

OSINT, and TECHINT. As a result of the significant analytic challenge involved a growing body of

interpretative techniques and methodologies has developed over many centuries. In the sense that

intelligence analysis is essentially concerned with the search for truth and meaning (albeit in particular

geopolitical, security and various specific contexts) and the communication of resultant insights to

decision-makers, then there are correlations to be drawn with the sub-disciplines of biblical exegesis and

hermeneutics and the overarching theological discipline they inform. Particular challenges that must be

addressed in biblical interpretation include: (1) the challenge of distance of time – the key events and

communications to be understood are separated from the contemporary interpreter by a vast gulf of time;

(2) the challenge of cultural distance – key actors and events reside within an ancient agrarian, middle

eastern context; (3) the challenge of geographical distance – requires an appreciation of the geography of

the Middle East at various points in history; (4) the challenge of linguistic distance – communication

occurs in the languages of Hebrew, Aramaic and Greek in the form those languages took thousands of

years ago; and (5) • various challenges around authorship and sourcing. The successful exegetical

practitioner, like the proficient intelligence analyst, must: (a) • understand the world behind the

communication being researched – the context of the source and the subject the source is reporting on; (b)

Understand the world within that communication – the language and structuring of the communication,

what is the source saying and how is the source saying it, and what meaning does this convey?; and (c)

understand the world before the communication – the context of the message recipient (the intelligence

consumer if you like). This paper will explore and discuss various areas of similarity and dissimilarity

between intelligence analysis practice and exegetical practice and will suggest some opportunities for

productive crossdisciplinary learning for intelligence practitioners.

Mandeep Dhami (University of Surrey): Intelligence Analysis and Decision Science

(Putting the Science into the Art of Intelligence Analysis) What is the probability that Iran will have nuclear weapons in 2012? What is the risk posed by

Anonymous to UK cyber space? Which al-Qaeda operatives should be prioritized for further intelligence

gathering? These are some of the types of questions that intelligence analysts must consider. The field of

Decision Science has existed for over 60 years, and is devoted to understanding how people perform

some of the sorts of tasks that intelligence analysts are faced with. For example, decision scientists ask:

How do people judge probability, perceive risk, and make choices? How do people search and select

relevant information, and use (weight and integrate) it appropriately when making decisions? Decision

scientists with backgrounds in disciplines such as Psychology and fields such as Behavioral Economics

develop and test correspondence- or coherence-based theories. They employ quantitative methods. And,

they consider normative, descriptive and prescriptive questions. I shall discuss how past research from the

field of Decision Science can be used to shed some light on how intelligence analysts might perform as

well as how their performance can be enhanced. In addition, I explore how theories and methods from

this field can be usefully applied to understanding the art of intelligence analysis.

16

Rhiannon Gainor and France Bouthillier (McGill University): Measuring Intelligence

Success: Conceptual and Methodological Challenges of Intelligence Measurement The purpose of intelligence practices is to source and develop intelligence to inform a decision. While there is a

significant body of literature dealing on how to implement and design intelligence processes, little is known

regarding how intelligence makes a difference in organizations. Measurement allows organizations to identify

deviations in standards, benchmark success, and whether objectives are met. When an expert, equipped with

intelligence, advises a decision-maker, it can be difficult to measure not only the role the intelligence played in the

decision, but the success of the decision. While intelligence processes can be and are measured with some effort,

identification and valuation of intelligence outcomes and impacts can be elusive. The fields of

competitive/business/strategic intelligence, intellectual capital, and knowledge management struggle with the

challenge of how to measure what is intangible, subjective, and frequently delayed in manifestation, as does the

larger intelligence field. Many organizations measure process, inputs, and satisfaction, and sidestep the considerable

methodological challenges related to accurate measurement of intelligence outcomes and impacts. The literature of

intangibles measurement provides some insights applicable to intelligence measurement, such as identifying

standard practice, problems with conceptualization and current measurement models, and best practices in

developing measures. The purpose of this paper is first, to clarify terms and concepts related to measurement,

asking: what is measurement, why measurement is needed, and what are the characteristics of ‘good’ measurement,

all within the context of measuring intangible outcomes. The second purpose of this paper is to identify from the

literature of intangibles measurement best practices and frameworks for developing measures of intelligence success

that account not only for costs and processes, but also outcomes and impacts. The intent is to identify conceptual and

methodological issues related to measurement and to offer for consideration insights from various fields that should

inform measurement strategies in intelligence.

WJR Gardner (Naval Historical Branch, UK Ministry of Defence): The Unreliable

Memoirs of an Applied Historian … and Operational Analyst and … The sound practices of history and intelligence have much in common and, it can be argued, each discipline can

learn from the other. But there may, in fact, be an even larger range of disciplines which may not be quite so

obvious or indeed make quite such a clear contribution to proper intelligence method and analysis. Arguably these

might range from art history to zoology; from paleontology to psychology. In the relatively restricted compass of a

conference paper or presentation it is not possible to produce a full description far less an evaluation of all these

potential relationships. However, this paper will attempt to take some consideration of a number of other disciplines

whilst concentrating on the symbiotic relationships that exist between intelligence and history, especially as it has

related to the experience of one historian and intelligence officer. These would range from consideration of sources,

analytical methods, understanding the significance of derived reporting and – perhaps as importantly - its limitations

and, lastly, the essential skills needed in growing and maintaining an effective relationship between producer and

consumer. It will be demonstrated that not only are there theoretical advantages to be gained on both sides but there

is also a considerable track record of practitioners, moving relatively seamlessly and successfully from one

discipline to the other, and also how many of the theoretical and analytical practices of one field can be applied to

the other.

Melissa Graves (University of Mississippi): Intelligence Analysis and Historiography Historians function similar to intelligence analysts. They consume huge amounts of disparate and oftentimes

questionably reliable material. Both must piece together incomplete stories. With both intelligence and history, a

source can make or break one’s ability to compose good analysis. Historians and analysts both construct their

particular version of world based upon primary records. Likewise, they face inherent perils of analytic biases,

particularly mirror imaging. Both understand that events and people do not exist within a vacuum and they must

accurately and fully take into account many competing forces. This presentation will discuss how the use of

historical methodology can assist analysts in looking at the broader picture and understanding trends and

motivations. In each story they construct, historians examine how race, class, gender, and religion affect key

players’ decisions. By looking at historical methodology, analysts can glean tactics helpful to their own work.

17

Nick Hare (UK Ministry of Defence): Intelligence Analysis and Economics At first glance, the overlap between 'economics' and 'intelligence analysis' should be significant. Both

disciplines are concerned with using evidence to test hypotheses about behaviour, and with modelling of

the actions of individuals and systems through examination of incentives at both micro- and macroscopic

levels. Why, then, is there relatively little practical interaction and overlap between the two? Based on an

informal survey of the few individuals who, in the UK, have worked in the intersection, I will present

some key similarities and differences between the professions of economics and intelligence analysis. I

will propose some tentative hypotheses to explain these similarities and differences, and identify some

key insights that each has to offer the other.

Michael Herman (Oxford University): Intelligence and Government Statistics British intelligence and statistics have evolved as government institutions over a similar periods as part of

the ‘knowledgeable state’, and useful comparisons can be drawn between their institutions. Despite being

pulled between centralization and decentralization, both have moved towards systems with central output

and influence on standards; but the moves are uneven. Statistics now has a national statistician and

supervisory authority overseeing what is still a mixture of central and departmental units: by contrast

intelligence’s professional authority has remained its central committee system. Arguably statistics has

moved in a natural direction for government’s knowledge producers, while British intelligence retains the

committee system formed as part of the tri-service military structure with which it fought the Second War

and moved into peace.

Carl Jensen (University of Mississippi): Intelligence Analysis and Futures Methodology The National Commission on Terrorist Attacks Upon the United States (also known as the 9-11

Commission) concluded that the U.S. intelligence community exhibited a “lack of imagination” in failing

to prevent the terrorist attacks of 2001. Like intelligence analysts, futurists are concerned with forecasting

future events; likewise, they must exhibit considerable imagination in divining emerging or distant trends.

Over the years, futures research has produced strategies for improving creative thinking. As well,

professional futurists have had to learn how to adapt their creative forecasts so that they have utility for

concrete and pragmatic customers. This presentation will discuss how the methods of futurists can be

integrated into the intelligence world, both in terms of analysis and establishing and promoting dialogue.

John Kringen (former CIA Director of Intelligence; Institute for Defence Analysis):

Judging Intelligence Success and Failure

There has been little public dialogue about the appropriate standards for evaluating the performance of

intelligence organizations. In the United States, for example, intelligence performance has typically been

judged on a case-by-case basis in the wake of perceived intelligence “failures” -- with limited discussion

or debate on the standards being applied. What are realistic standards for evaluating the performance of

intelligence organizations in an uncertain world? At a minimum, they need to go beyond the notions of

surprise and embarrassment. What responsibilities do customers and those entities responsible for

oversight of intelligence have for the performance of intelligence organizations? In this regard, clear

guidance on such matters as intelligence priorities is critical. Finally, what insights on appropriate

performance standards can be gained from examining practices outside the intelligence enterprise? For

example, if major defense contractors in Europe and the United States are often challenged in terms of

their ability to project their own delivery schedules, what standards should be applied to weapons analysts

in the intelligence community? This PowerPoint presentation will propose a framework for evaluating the

performance of intelligence organizations – drawing upon debates in recent decades about “intelligence

failure” and relevant practice in other professions

18

William Lawhead (University of Mississippi): Seeing Is Not Believing: Insights for Intelligence Analysis from

Professional Magicians. To relate magic (conjuring) and intelligence analysis seems like quite a stretch. However,

throughout history magicians have served their country by using their professional skills in warfare and espionage. I

briefly mention some examples of how magicians have used their knowledge to make contributions to these two activities.

But these two activities are not the same as analysis. What do the skills and knowledge of professional magicians have to

offer to intelligence analysts? This paper will suggest that they have a lot to teach us in the areas of deception, perception,

and cognition. Deception The most obvious application of magic to intelligence analysis is in the area of deception and

counterdeception. Here, the goals of the magician and the analyst are opposite. The magician tries to deceive us and the

analyst tries to avoid being deceived or to uncover attempts at deception. But, understanding the magician’s techniques

and principles will be useful in understanding deception in the intelligence arena. Barton Whaley is a political scientist

who is a leading authority and author on the topic of military deception. He is also a magician and author of several books

on magic. Some of his insights, based on knowledge of magic, will be discussed. Perception A greater knowledge of how

we experience the world will be useful to the analyst. Here, magicians can instruct us. After all, magicians are able to

make us think we see things that aren’t there and make us not see things that are there. Recently, neuroscientists have

started doing research on why magicians fool us. For example, scientists have used eye-tracking technology on subjects

while they watched a magician perform in order to see what causes the illusion. They found that the magician’s use of

misdirection did not divert the gaze of the audience but diverted their attention. This is related to the phenomenon of

“inattentional blindness.” We do not see objects in our visual field (including the magician’s sneaky moves) even though

photons from the object are reaching our retina, because of a manipulation of our attention. Certainly, for the analyst,

staring at lots and lots of data is not enough. We have to be concerned with how the data is being viewed and how our

attention is directed or misdirected. Cognition Related to the previous topics is the issue of how the magician controls our

minds. While some people think that misdirection is the main tool of the magician, other theorists think it is the way that

the magician plants assumptions in our minds or exploits the assumptions we already have. Magician Roberto Giobbi says

“Magic should be very easy, since our spectators fool themselves. All you need to do is avoid any words, thoughts or

actions that interrupt this tendency.” Since assumption checking is an important technique for the analyst, an examination

of how magicians use our assumptions will be instructive.

Stephen Marrin (Brunel University): Learning from Other Disciplines, including Medicine

and Journalism Revisits and extends call for research agenda articulated at the end of the 2005 article “Improving Intelligence

Analysis by Looking to the Medical Profession”: Crossing Professional Lines: Finally, the lessons that intelligence

can draw from an examination of the similarities and differences with the medical profession indicate the

importance of looking to analogous professions for ideas that can be adapted to an intelligence context. Doing so

might help improve finished intelligence production processes and the incorporation of intelligence into

decisionmaking. Analogies serve a number of purposes, such as aiding communication about difficult topics by

finding illustrative examples in other fields, or by more directly affecting existing ways of doing business through

the incorporation of tools that exist to achieve similar purposes in other fields. Many of the challenges intelligence

analysts face are not as unique as its practitioners believe, but the insularity of the field prevents them from being

able to identify the lessons from other professions that could be useful as models to follow. As a result, the first task

is to identify analogous professions, and examine them for the lessons they might provide. Any profession that

encounters similar problems-such as medicine, journalism, law, or law enforcement- may provide fertile ground for

deriving ideas to improve existing practices. Perhaps if intelligence analysts adopted methods from analogous

professions-or adapted them to the unique requirements of intelligence analysis-some of the obstacles they currently

face in accurately portraying their understandings of the international environment could be overcome.” (Source:

Stephen Marrin and Jonathan Clemente. Improving Intelligence Analysis by Looking to the Medical Profession.

International Journal of Intelligence and CounterIntelligence, 18:(707-729). 2005 (pps 726-727).

19

Stephen Marrin (Brunel University): Evaluating the Quality of Intelligence Analysis: By

What (Mis) Measure?: Each of the criteria most frequently used to evaluate the quality of intelligence

analysis has limitations and problems. When accuracy and surprise are employed as absolute standards, their use

reflects unrealistic expectations of perfection and omniscience. Scholars have adjusted by exploring the use of a

relative standard consisting of the ratio of success to failure, most frequently illustrated using the batting average

analogy from baseball. Unfortunately even this relative standard is flawed in that there is no way to determine

either what the batting average is or should be. Finally, a standard based on the decisionmakers’ perspective is

sometimes used to evaluate the analytic product’s relevance and utility. But this metric, too, has significant

limitations. In the end, there is no consensus as to which is the best criteria to use in evaluating analytic quality,

reflecting the lack of consensus as to what the actual purpose of intelligence analysis is or should be.

Peter Nardulli and Dan Roth (University of Illinois): Advancing our Understanding of

Global Patterns of Civil Unrest: An Interdisciplinary Effort This paper deals with an area of increasing concern to intelligence analysis: the study of civil unrest, including

destabilizing acts initiated by private citizens and disruptive state actions. It is based on one component (the Societal

Stability Protocol, or SSP) of a six-year multidisciplinary effort, the Social, Political and Economic Event Database

project (SPEED), which was initiated by and housed in the Cline Center for Democracy. The paper will outline how

SPEED has used diverse technologies to enhance human capacities to conduct core intelligence functions: the

detection, documentation, summarization and analysis of destabilizing developments. The paper will consist of

three parts. The first will briefly outline: (1) the creation of SPEED’s global news archive, which contains over 150

million digitized news reports from a diverse set of news sources for every country in the world for the post-1945

era; (2) the creation of its destabilizing event ontology, which serves to focus the project’s detection efforts; and (3)

its use of automatic text classification and event extraction technologies to capture pertinent information within the

news archive. The second main section will present preliminary results for a tool developed by Roth’s Cognitive

Computation Group, the Event Annotation Tool (EAT+). EAT+ uses advanced NLP technologies in conjunction

with large amounts of training data to annotate key textual passages (trigger words, actors, dates, places, etc.). The

technology underlying EAT+ will be outlined and success rates for detecting references to a representative set of key

civil unrest events (generic political attacks, kidnappings, assassinations, assembly of coercive forces, imposing a

curfew, declaring a state of emergency, restricting movement of citizens, etc.) will be reported. The final section of

the paper will report on SPEED’s efforts to harness the analytic power of SSP data through the use of various

composite measures of event intensity and event origins. This section will outline post WWII trends in civil unrest

and state repression.

Sean Newman (Fire Department of New York): How Adherence to Journalistic Standards

Positioned FDNY Watchline at the Forefront of Fire Service Analytical Intelligence Captain Newman’s presentation will show why the FDNY*s weekly intelligence newsletter, Watchline,

continues to add readers and garner praise from across the globe by following some basic rules of

journalism, such as concise headlines and lead sentences, brevity, story selection, and the collective

editing process. Other topics to be discussed are feature stories, deadlines and the importance of feedback.

Also, attendees will see how Watchline compares to some of Sherman Kent’ principles.

20

Stefania Paladini (Coventry University): Bayesian analysis in intelligence and social

sciences: a tool for all trades?” Bayesian analysis is one of the most versatile methods in social

sciences, with its application spanning from probability to finance to psychology. Incidentally, it is also

one which relevance in intelligence is well known and documented, at least since Sherman Kent’s times.

In a now quite famous text, Psychology of Intelligence Analysis, Heuer also provided some interesting

examples of its potential and its worthiness for the job. Also, he argued with reason, Bayesian reasoning

can significantly reduce the effects of cognitive biases, one of the main problems in intelligence. Still,

Bayesian analysis’ application, in intelligence as much as in social sciences, can be somehow problematic

or difficult, for a series of reasons – first of all, challenges linked to its technical complexity. It has been

proved that even people mathematically trained are generally poor in estimating conditional probabilities,

which are the objects of Bayes’ Theorem. Furthermore, some of its difficulties are linked to the quality of

the prior information available, as well as to the collection of suitable posterior ones. However, by

reviewing the available literature and by conducting some experiments in this sense it is the author's

impression that the instrument fails more than often due to the fact that it is used to address the wrong

questions - and not because of its internal weaknesses. The aim of this paper is to explore the issues

related to Bayesian analysis, to define when its use is appropriate and in which term, both in intelligence

and in social sciences and to provide suitable evidence.

Michael Robinson (Deloitte and Touche, LLP): Intelligence Analysis and Digital Forensics There are a large number of similarities between the fields of intelligence analysis and digital forensics.

Both disciplines collect data from a variety of sources, analyze data to form actionable intelligence, and

report information to stakeholders in an unbiased and timely manner. In the field of digital forensics,

analysis results in the formation of a report that can withstand legal scrutiny and be incorporated in legal

proceedings/trials. While there are similarities in the goals and objectives of the two disciplines, there are

resemblances in the current obstacles as well. Many of the challenges encountered in digital forensics are

also occurring in intelligence analysis, for example, there is a need to collect and process large amounts of

data (often terabytes) in a short amounts of time, there are a limited number of seasoned practitioners in

the field, there is a finite pool of resources, there is a need to provide qualitative indicators of reliability

and confidence to analyzed data, there is a need to provide information in an easily digestible format for

stakeholders that have limited time to read them, and there is a need to overcome misinformation that

appears from anti-forensics techniques. This paper/presentation will examine some of the subspecialties

within digital forensics, such as media analysis, mobile device forensics, and intrusion analysis, to

identify current obstacles and current solutions to these mounting problems. Comparisons and contrasts

between the fields of intelligence analysis and digital forensics will be made with an emphasis on

introducing solutions to overcome the current hurdles.

Patricia Rooney (National Ground Intelligence Center, US Army) and Peter Ronayne

(Federal Executive Institute): Neuroscience in Intelligence Analysis: Understanding How

the Brain Works and Its Impact on Analytic Decision Making: Intelligence analysis is neither

straightforward nor foolproof; it is both an art and a science. Whether pursuing strategic warning, current

intelligence or estimative intelligence, all analysts depend upon neuro-psychological capabilities that are

in constant tension and competition with one another. Most analysts know little about this neuro-tension

or the neurological processes that leave analysts vulnerable to deception in their daily analyses. This

session provides participants with (1) an understanding of the neurological tensions and cognitive

plasticity that analysts can draw upon for effective decision making; (2) key cognitive fitness tools to

keep the decision-making apparatus in tip-top shape; and (3) an understanding of intuition (art) and

rational, methodical (science) reasoning for analytic conclusions.

21

Gabriel Sebe (Bucharest University): Political Marketing Intelligence The aim of the present paper is to study the interaction in conceptual and applied terms between the

intelligence realm and the political marketing field; it is organized in four sections. The first section takes

into account two dimensions. The first focus on the succession series interaction between intelligence and

other fields conceptually related with the political marketing realm: journalism (Park, Lippmann, etc.),

public relations (Ivy Lee, Bernays, etc.), political science (Shotwell, Lasswell, etc.), international relations

(Shotwell, Mitrany, etc.), etc. The shift induced by Vannevar Bush’s paradigm and Shannon’s

information theory allows an integration, imposing a conceptual platform related with the informational

society concept; this is the second dimension. Now an explicit relationship between Levitt’s marketing

myopia, Kotler’s marketing concept broadening, Kotler and Singh marketing warfare setting, Newman

and Sheth political marketing mapping, Hunt’s resource/advantage theory and some main issues from the

intelligence field follows. Putting Kent’s contribution in this framework and taking into account the

Kendall/Kent debate, someone infers new perspectives on the knowledge transfer problem in terms of

interdisciplinarity, multidisciplinarity and transdisciplinarity. The second section introduces a

segmentation compatible with Downs’ political information and Machlup’s knowledge production,

adequated in terms of the intelligence cycle idea Boyd’s OODA framework. In this manner it becomes

possible to take into account two process-like dimensions. First the consideration of deliberation/decision

processes related with BI, CI, SI and their corresponding counterparts from the political marketing field.

Secondly, the differences and similarities concerning the meaning of ‚analysis’ as usable knowledge

transfer and fusion knowledge dedicated to support a competitive decision process. Beyond

terminological labels, the source classification problem, the relevance criteria, the measurement of

meaning paradigms, the transparency/opacity policy, the secrecy patterns, etc. and their associated

processes in terms of head building through education, instruction, trening form the basis of the analogies

necessary for conceptual transfer between these realms. In fact many problems treated in intelligence

terms can’t be conceived without a political marketing representation; the international relations

trade/conflict problem (Crucé, 1623; Polachek, 1980; etc.) is one example. Both intelligence and political

marketing fields are transdisciplinary; they cannot replace each other however together in a controllable

and rational knowledge fusion setting allow better solutions. The third section maps the analogies

between the intelligence field components and political marketing structured processes. Putting in

evidence some micro/macro principles it becomes possible to relate the specific knowledge

representations of the intelligence market conceived as the environment for both an intelligence economy

and a competitive political marketplace. Taking into account some preceding considerations, a certain

segmentation of research (fundamental, applied, practical, commercial) is put in correspondence with

distinct manners of learning generating different intellectual profiles in terms of theoretical/practical

capabilities. In this manner we enlarge the meaning for both usable knowledge and knowledge transfer

between theory and practice not only in conceptual, but also in human resource terms, giving a meaning

for the concept of intelligence capital. The fourth section concludes on the relevance of the ‚political

marketing intelligence’ paradigm from Boyd’s destruction and creation perspective.

22

Musa Tuzuner (Intelligence Studies Research Center; Turkish National Police Academy):

Multi-Level Cognitive Biases in the Chain of Intelligence Flow—From Source to

Consumers: Improving intelligence analysis has been attracting more attention in the last decade

because of the significant intelligence failure occurring in the world. Very few scholars have examined

the cognitive bias factors’ negative affects on the intelligence analysis. They believe that understanding

cognitive biases affecting intelligence biases would better serve to have efficient intelligence product for

the consumers. However, in order to produce better intelligence product, we need to understand the

Cognitive biases, not only in the intelligence analysis, but also in the chain of Intelligence flow. From our

intelligence field experience, we have found a flaw in looking for cognitive bias in the intelligence

analysis (Evaluators). By the time intelligence information reaches the analysis level, a cognitive bias is

already embedded when going from Informers to Collectors and so on. This is why detecting the

cognitive biases in the Evaluators stage is not adequate to produce efficient intelligence for the

consumers. In order to find where the biases originate, we have created a “Manual for Detecting

Intelligence Failure”. In order to detect multi-level cognitive biases in the chain of intelligence flow

Aydinli and I are collecting data (through interviews) from intelligence practitioners in Pakistan, Turkey,

Iraq, and Lebanon. After analyzing this dataset, we will classify the cognitive origins of the defects in the

intelligence flow: as individual origins of defects and institutional origins of the defects. Then we will

discuss how these collection, evaluation, and sharing defects can lead to intelligence failure.

Kathleen Vogel (Cornell University): Current Intelligence Reporting and the H5N1 Bird

Flu Virus: Insights from the Field of Science and Technology Studies In late 2011, virologists Ron Fouchier and Yoshihiro Kawaoka encountered a swarm of government and

public controversy from their creation of novel variants of the H5N1 bird flu virus. Prior to publication of

these experimental findings, the U.S. National Science Advisory Board for Biosecurity (NSABB)

reviewed Fouchier’s and Kawaoka’s scientific manuscripts. The NSABB unanimously recommended that

the, “conclusions of the manuscripts be published but without experimental details and mutation data that

would enable replication of the experiments.” The NSABB explained its justification as being based on

security concerns: “publishing these experiments in detail would provide information to some person,

organization, or government that would help them to develop similar viruses for harmful purposes.”

However, once the NSABB made its decision public, other flu virologists emphasized the difficulty of the

experiments and how these experimental results could lead to beneficial new medical treatments. Other

scientists, however, have sided with the NSABB and warned of the security dangers in publishing these

bird flu results. In the middle of this context, U.S. intelligence analysts have been tasked by their

superiors to provide up-to-date security assessments of these bird flu experiments as new information

about the experiments continues to emerge. These intelligence analysts have struggled to make sense of

the positions held by the NSABB and different scientific experts. These analysts have also encountered

challenges with how to gather and evaluate information in a timely fashion to better assess the role of

explicit and tacit (know-how) knowledge in these experiments. These intelligence assessments are needed

to better inform policymakers about whether to censor future logical research findings and how to think

about emerging biological science threats. This paper will discuss what happens when academic scholars

from the social science field of science and technology studies (S&TS) are brought into unclassified

dialogues with U.S. intelligence analysts as these analysts are writing their current intelligence reports on

these controversial bird flu experiments. This paper will illustrate how S&TS research can provide new

insights for assessing the security threat from these experiments and related biosecurity/WMD concerns,

how S&TS research can more broadly inform the analytic practices that underpin current intelligence

reporting, as well as challenges that exist in bringing S&TS ideas to bear on intelligence analysis.