forecasting conflict lecture 1 technical political...

Forecasting ConflictLecture 1

Technical Political Forecasting: An Overview

Philip A. Schrodt

Parus Analytical [email protected]

Graduate School of Decision SciencesUniversity of Konstanz14 - 17 October 2013

Overview of the series of lectures

1. Prediction: contemporary opportunities and challenges

2. Conflict forecasting in U.S. government projects

3. Event data and GDELT

4. Forecasting: Conventional time series approaches

5. Forecasting: Sequence and clustering approaches

Overview for First Lecture

I Justification of prediction

I Seven opportunities

I Seven challenges

Approach of the lectures

I Breadth, not depth!—this is more of a “bird’s eye view”I (But not—I repeat, not!—a “god’s eye view”!)

I A guide to vocabulary[ies], approaches and what you need toknow

I you can then follow up on all of this material in detail. If I canlook it up, you can look it up

I Emphasis on practical applications: Some of the slides arerecycled from presentations I’ve given in the U.S. policycommunity

I This is a feature, not a bug

I This is the departure lounge, not the baggage claimI All of the slides are available at

http://eventdata.parusanalytics.com/presentations.html

The Debate

Whatever is not forbidden is mandatory

The old challenge: “Prediction is not scientific”

Huh????

The new challenge: “Political science (and economics) is notscientific because it can’t predict”

I See various op-eds in New York Times and related venues overthe past eighteen months. Despite Nate Silver’s predictions.

I See complete suspension of US NSF Political Science programI +<4> The sting of which has been somewhat mollified by

subsequent efforts by the same people to suspend the entire USgovernment

Whatever is not forbidden is mandatory

The old challenge: “Prediction is not scientific”Huh????

The new challenge: “Political science (and economics) is notscientific because it can’t predict”

I See various op-eds in New York Times and related venues overthe past eighteen months. Despite Nate Silver’s predictions.

I See complete suspension of US NSF Political Science programI +<4> The sting of which has been somewhat mollified by

subsequent efforts by the same people to suspend the entire USgovernment

Problems with qualitative approaches

Tetlock: Experts typically do about as well as a “dart-throwingchimp”

Except for television pundits, who do even worse. Ask PresidentRomney.The media want things to be dramatic. “We’re all going to die!Details follow American Idol”

Qualitative theory isn’t much better:Remember the hegemonic US seizure of undefended Canadian andMexican oil fields in response to the 1973 OPEC oil embargo?

SMEs and the “narrative fallacy”

SME = “subject matter expert”

Hegel: the owl of Minerva flies only at dusk

Taleb (Black Swan): seeking out narratives is an almost unavoidablecognitive function and it generates a dopamine hit

This is your brain on narratives

Problems with quantitative approaches

Ward, Greenhill and Bakke (2010): Models based on significancetests don’t predict well because that is not what a significance test issupposed to do.

Gill, Jeff. 1999. The Insignificance of Null Hypothesis SignificanceTesting. Political Research Quarterly 52:3, 647-674.

The norm in political science has been to do full-sample evaluation,whereas the norm in machine-learning has been split-sample, which isusually more robust and is certainly more credible

Role of prediction for logical positivistsHemple: “Explanation” in the absence of prediction is “prescientific”

I Critical case: astrology vs astronomyI More generally, mythological accounts provide “explanation”

[Quine]

Prediction was simply assumed to be a defining characteristic of agood theory until relatively recently

Arguably, no philosopher of science prior to the mid-20th centurywould find the frequentist-based “explanation” emphasized incontemporary political science even remotely justified

I Leaving aside that frequentism is logically inconsistent and hasbeen characterized in Meehl (1978) as “a terrible mistake,basically unsound, poor scientific strategy, and one of the worstthings that ever happened in the history of psychology”

I Hey, dude, tell us what you really think. . .I But that is another lecture. . .

Explanation, continuedPhilosophers of science have long suspected that it is possible to havea seemingly sound explanation of a phenomenon that confers nopredictive leverage over the phenomenon (Nagel, 1961; Toulmin,1961). For instance, plate tectonics theory is the received explanationfor earthquakes, but it confers no ability to generate accuratepredictions about when earthquakes will occur. Conversely, it ispossible to have remarkable predictive accuracy that rests on a deeplyflawed framework. Ancient astronomers generated predictivelypowerful celestial charts even though they didn’t have the faintest ideawhat planets or stars were.. . .How patient should we be with low-predictive-accuracy theories?When should we tune out the theorists and go with algorithms that nomore understand world politics than ancient astronomers understoodcelestial motion? We have no off-the-shelf answer, but we resonate toLakatos’s (1970) rule for distinguishing degenerative fromprogressive research programs: forgive patch-up operations only ifthey inspire testable propositions that pan out. [Tetlock, 2013]

Additional issues in explanation vs. theory

Hume: the problem of inductionI Farmer’s cat vs. farmer’s turkey

Friedman: unreasonable assumptions are justified provided thepredictions are accurate

I Justification for rational choice modelsI Issue: the “provided predictions are accurate” part tends to be

forgotten, and is far too often replaced with “provided I think theassumptions are elegant and/or make my life easier”

Success without theory: Gothic cathedrals

Note that these issues affect observational studies but notexperimental studies, which is why experiments are used wheneverpossible.

Kahneman et al: people are really bad at statisticalreasoning

I Everyone, including statisticians unless they focus very hard

I Example: managed mutual funds, which both theory andevidence indicate cannot work

I Example: opposition to “evidence based medicine” in the US,with a preference for clinical intuition even when this has beendemonstrated to be less effective

I Probabilitistic weather forecasts seem to be the one majorexception: rain likelihood, hurricane tracks

The Necessity of Prediction in Policy

Feedforward: policy choices must be made in the present foroutcomes which may not occur for many years

Planning Times: even responses to current conditions may requirelead times of weeks or months

[More on this tomorrow]

“Schrodt should do everything in ‘sevens”’

Opportunities

I Totalitarian law of the universe: whatever is not forbidden ismandatory. Prediction is scientific [now]

I Data: small, big, fastI You can’t solve everything with more machine cycles, but it

never rarely hurtsI Successful large-scale projects: PITF, ICEWS, ACE, ENCoReI (Mostly) Convergent modelsI Location, location, locationI Open source, open access, open collaboration

Challenges

I Determining credible metricsI Black swansI Heterogeneous environmentsI Absence of theories indicating what is and is not predictableI Pournelle’s Law: no task is so virtuous that it will not attract

idiotsI Ethical concerns

Multi-disciplinary challenges

Big Data: Machine learning researchers routinely use social sciencedata to construct models. Many of these achieve high accuracy insplit-sample tests, to the point where these researchers simply assumethat things are predictable.

IARPA ACE “Good Judgment Project” (Tetlock): While mostforecasters do no better than chance, a small number of “superforecasters” perform significantly over long periods of time and largenumbers of questions. Furthermore these individuals exhibit commoncharacteristics and strategies, and to a limited extent, these can betaught. Forthcoming article in Economist: The Year 2014

Early work (1980s) in “expert systems” for classification problemsshowed the systems tended to be about 10% more accurate thanhumans and achieved this accuracy using less information.

The Forecaster’s Quartet

I Nassem Nicholas Taleb. The Black Swan(most entertaining obnoxious)

I Daniel Kahneman. Thinking Fast and Slow(30 years of research which won Nobel Prize)

I Philip Tetlock. Expert Political Judgment(most directly relevant)

I Nate Silver. The Signal and the Noise(high level of credibility after perfect 2012 electoral votepredictions)

Though this may be going a little far...

Computing power

Computationally-intensive methods

I Bayesian estimation using Markov chain Monte Carlo methods

I Bayesian model averaging (“AJPS-as-algorithm”)

I random forest models

I large-scale textual databases

I machine translation

I geospatial visualization

I real-time automated coding

I remote sensing data such as nightlight density

Large Scale Conflict Forecasting Projects

I State Failures Project 1994-2001I Joint Warfare Analysis Center 1997I FEWER [Davies and Gurr 1998]I Center for Army Analysis 2002-2005I Swiss Peace Foundation FAST 2000-2008I Political Instability Task Force 2002-presentI DARPA ICEWS 2007-presentI IARPA ACE and OSII Peace Research Center Oslo (PRIO) and Uppsala University

UCDP models

(much more on this tomorrow)

Convergent ResultsI Most models require only a [very] small number of variablesI Indirect indicators—famously, infant mortality rate as an indicator of

development—are very usefulI Temporal autoregressive effects are huge: the challenge is predicting

onsets and cessations, not continuationsI Spatial autoregressive effects—“bad neighborhoods”—are also hugeI Multiple modeling approaches generally converge to similar accuracyI 80% accuracy—in the sense of AUC around 0.8— in the 6 to 24 month

forecasting window occurs with remarkable consistency: few if anyreplicable models exceed this, and models below that level can usuallybe improved

I Measurement error on many of the dependent variables—for examplecasualties, coup attempts—is still very large

I Forecast accuracy does not decline very rapidly with increased forecastwindows, suggesting long term structural factors rather than short-term“triggers” are dominant. Trigger models more generally do poorlyexcept as post hoc “explanations.”

Location, location, location!

ACLED Geospatial

UCDP Geospatial

GDELT: Afghanistan, District-level Violence

Source: Jay Yonamine and Joshua Stevens, Penn State

GDELT: Cairo protests

Source: David Masad and Andrew Halterman of Caerus Analytics.

Open source, open access, open collaboration

I There is a strong if incomplete norm towards open sharing ofdata and methods

I Unintended consequence: PITF “forecasting tournament” cannotbe published in a major journal because it used proprietarydata—the baseline data has 2,700 variables—that cannot bearchived in replication sets. The results are, however, stillavailable on SSRN.

I The inability to share source texts is clearly a concern innews-report-based datasets such as GDELT and MID.

I By all available evidence, US government forecasting projectsare using similar methodologies to those available in opensources; in fact they are probably lagging somewhat behind this

I We now have significant NGO and academic work, and aninternational “epistemic community” has developed around thetopic.

CHALLENGES

Metrics

What is being predicted

I Probability of binary outcomes by a fixed date

I Quintile rankings of risk / probability-based “watch lists”

I Survival and hazard models

I Switching and phase models

I Networking—both social and geospatial—models

All of these can be used as input to ensemble methods

Classification Matrix

ROC Curve

Source: http://csb.stanford.edu/class/public/lectures/lec4/Lecture6/Data_Visualization/images/Roc_Curve_Examples.jpg

Separation plots

And wait, there’s still more!

I Recall / True Positive Rate/Sensitivity

I Precision / Positive predictive value (PPV)

I Specificity / True Negative Rate

I F1 score: harmonic mean of precision and recall

I Beier scores

I Posterior probabilities

I Proportional reduction of error or entropy

Black swans

Ideal forecasting targets are neither too common nor too frequent

Good Judgment Project: look for events with a 10% probability

The Forecasting Zoo

Ducks can be interesting...

And this is going too far. . .DARPA-World!

By definition, most black swans will not occur! So there is little point ininvesting a large amount of effort trying to predict them.

“Can your model predict a chemical attack by self-recruited Mexican jihadisworking as rodeo clowns in Evanston, Wyoming? Why not?!"

And this is going too far. . .DARPA-World!

By definition, most black swans will not occur! So there is little point ininvesting a large amount of effort trying to predict them.“Can your model predict a chemical attack by self-recruited Mexican jihadisworking as rodeo clowns in Evanston, Wyoming? Why not?!"

Challenge: distinguishing black swans from rare eventsBlack swan: an event that has a low probability even conditional onother variables

Rare event: an event that occurs infrequently, but conditional on anappropriate set of variables, does not have a low probability

Medical analogy: certain rare forms of cancer appear to be highlycorrelated with specific rare genetic mutations. Conditioned on thosemutations, they are not black swans.

Another important category: high probability events which areignored. The “sub-prime mortgage crisis” was the result of the failureof a large number of mortgage which models had completelyaccurately identified as “sub-prime” and thus likely to fail. This wasnot a low probability event.Upton Sinclair: It is hard to persuade someone to believe somethingwhen he can make a great deal of money not believing it.

Heterogeneous environmentsI Per Pinker, Goldstein, Mueller, etc, is the system changing

significantly while we are trying to model it? How far back aredata still relevant?

I How different are various types of militarized non-state actors?For example, how much do al-Qaeda and international narcoticsnetworks have in common?

I We are also using a more heterogenous set of forecastingmethods, and probably do not understand their weak points aswell as we understand those of regression-based models.

I Threats tend to occur in small number of “hot-spots”I Europe 1910-1945I Middle East 1965-presentI Balkans in 1990sI Internal conflict in India

Note that all of these are complicated by rare events—some of whichmay be black swans—since it limits the number of observations wehave on the dependent variable.

Changing nature of conflictThreats in 1910I “Gunboat diplomacy” was an accepted norm, as were elements

of bellicose and social DarwinismI Some competition occurred between approximate equalsI Mediation was ad hoc with no established international

institutionsI Territorial change was credible

Threats in 2010I Highly asymmetric distribution of military powerI Threats get almost immediate attention from potential mediators,

including the UNI Non-military sanctions are credible (Iraq, Iran)I Territorial changes are rare and highly problematic

Will changes in the technological environment—internet, UAVs,various monitoring technologies—change probabilities?

Theory: what can and cannot bepredicted?

Is astronomy scientific?

Astronomy generally has a very good record of prediction, and fromthe earliest days of astronomy, successful prediction has been a keylegitimating factor

I relation between star positions and events like the Nile floodI eclipsesI orbitsI Halley’s cometI precision steering of space-craft

Nonetheless, astronomy cannot predict, nor does it attempt to predict:I solar flares, despite their potentially huge economic

consequencesI previously unseen cometsI next nearby supernova

Determinism: The Pioneer spacecraft anomaly

“[Following 30 years of observations] When all known forces actingon the spacecraft are taken into consideration, a very small butunexplained force remains. It appears to cause a constant sunwardacceleration of (8.74 ± 1.33)× 10−10m/s2 for both spacecraft.”

Source: Wikipedia

Irreducible sources of error-1

I Specification error: no model of a complex, open system cancontain all of the relevant variables;

I Measurement error: with very few exceptions, variables willcontain some measurement error

I presupposing there is even agreement on what the “correct”measurement is in an ideal setting;

I Predictive accuracy is limited by the square root of measurementerror: in a bivariate model if your reliability is 80%, youraccuracy can’t be more than 90%

I This biases the coefficient estimates as well as the predictions

I Quasi-random structural error: Complex and chaoticdeterministic systems behave as if they were random under atleast some parameter combinations .Chaotic behavior can occur in equations as simple asxt+1 = axt

2 + bxt

Irreducible sources of error-2

I Rational randomness such as that predicted by mixed strategiesin zero-sum games

I Arational randomness attributable to free-willI Rule-of-thumb from our rat-running colleagues:

“A genetically standardized experimental animal, subjected tocarefully controlled stimuli in a laboratory setting, will dowhatever it wants.”

I Effective policy response:in at least some instances organizations will have taken steps tohead off a crisis that would have otherwise occurred.

I The effects of natural phenomenonI the 2004 Indian Ocean tsunami dramatically reduced violence in

the long-running conflict in Aceh

(Tetlock (2013) independently has an almost identical list of theirreducible sources of error.)

Open, complex systems

Balancing factors which make behavior predictable

I Individual preferences and expectations, which tend to changevery slowly

I Organizational and bureaucratic rules and norms

I Constraints of mass mobilization strategies

I Structural constraints:the Maldives will not respond to climate-induced sea level riseby building a naval fleet to conquer Singapore.

I Choices and strategies at Nash equilibrium points

I Autoregression (more a result than a cause)

I Network and contagion effects (same)

“History doesn’t repeat itself but it rhymes”Mark Twain (also occasionally attributed to Friedrich Nietzsche)

Paradox of political prediction

Political behaviors are generally highly incremental and vary littlefrom day to day, or even century to century (Putnam).

Nonetheless, we perceive politics as very unpredictable because wefocus on the unexpected (Kahneman).

Consequently the only “interesting” forecasts are those which areleast characteristic of the system as a whole. However, only some ofthose changes are actually predictable.

Finding a non-trivial forecast

I Too frequent: prediction is obvious without technical assistance

I Too infrequent: prediction may be correct, but the event is soinfrequent that

I The prediction is irrelevant to policyI Calibration can be very trickyI Accuracy of the model is difficult to assess

I “Just right”: these are situations where typical human accuracy islikely to be flawed, and consequently these could have a highpayoff, but there are not very many of them.

Differing attitudes towards error

Geography:I Progress consists of ever more accurate data

Political science:I Trust nothing—everything has error, just control for the

systematic biases

From presentation to a geospatial intelligence conference, where I suspect ithad zero impact. . .

The Political Science View of the World2 October 2012: Israeli Prime Minister Netanyahu reported to bepursuing increased sanctions against Iran

I He has changed his mind and thinks sanctions are an effectiveapproach(face value)

I He has concluded Obama will be re-elected(international considerations)

I The Israeli military finally persuaded him an attack was a badidea(domestic considerations)

I Israel is going to attack Iran in the near future(deception strategy)

(presumably apocryphal) exchange at the Congress of ViennaI Aide: Your excellency, the Russian Ambassador has just died!I Prince von Mitternich: Fascinating...now, what are his

intentions?

Differing attitudes towards error

Geography:I Progress consists of ever more accurate data

Political science:I Trust nothing—everything has error, just control for the

systematic biases

Machine learning::I it is what it is: goal is improving prediction

Statistics::I signal to noise: Perfect is the enemy of "good enough"I mathematically approximate the characteristics of the errorI Taleb, Mandelbrot: don’t be a Gaussian in a power-law world

Models matter

Arab Spring is an unprecedented product of the new social mediaI Model used by Chinese censors of NSM: King, Peng, Roberts

2012I Next likely candidates: Africa

Arab Spring is an example of an instability contagion/diffusionprocess

I Eastern Europe 1989-1991, OECD 1968, CSA 1859-1861,Europe 1848, Latin America 1820-1828

I Next likely candidates: Central Asia

Arab Spring is a black swanI There is no point in modeling black swans, you instead build

systems robust against them

Statistical and modeling challengesRare events

I Incorporate much longer historical time lines?—Schelling usedCaesar’s Gallic Wars to analyze nuclear deterrence

I New approaches made possible by computational advances

Analysis of event sequences, which are not a standard data typeI There are, however, a large number of available methods, and it

is just possible that these will work with very large data sets suchas GDELT

I This possibility will be discussed in detail in Lecture 5

CausalityI Oxford Handbook of Causation is 800 pages long

Integration of qualitative and qualitative/subject-matter-expert (SME)information

I Bayesian approaches using prior probabilities are promising butto date they have not really been used

Pournelle’s Law:No task is so virtuous that it will not attract idiots

I Need to establish with the media and policy-makers that notevery forecast, even especially those made using “Big Data”methods, is scientifically valid

I It took the survey research community about thirty to forty yearsto establish professional credibility, though they have largelysucceeded

I Conveying limitations of the methods against thehyper-confidence of pundits and individuals with secret models

I Limitations of the data sourcesI Limitations of the data coding, particularly automated codingI Limitations of the model estimationI Limitations of probabilistic forecasts, particularly for rare events,

even when the models are correct

Critical case: studies of climate change and conflict. As Pinker andGoldstein noted, people want to hear simple scary answers.

Ethical concerns

I Thus far, we’ve generally had the luxury of no one payingattention to any of our predictions : what if governments do startpaying attention?

I “Policy relevant forecast interval” is around 6 to 24 monthsI USAID/FAO famine forecasting modelI It is possible that our models could become less accurate because

crises are being averted, but I don’t see that happening any timesoon.

I Difficulties in getting anyone, including experts (see Kahneman,Tetlock), to correctly interpret probabilistic forecasts

I Possible impact on sourcesI Local collaboratorsI Journalists (cf. Mexico)I NGOs to the extent we are using their information

Thank you

Email: [email protected]

Slides: http://eventdata.parusanalytics.com/presentations.html

Forecasting papers:http://eventdata.parusanalytics.com/papers.html

forecasting conflict lecture 1 technical political...

Documents