geer - hutton - shannon - a pilot project on the use of prediction markets in information security

Dan Geer, In-Q-Tel Alex Hutton, Verizon Business

Greg Shannon, Carnegie Mellon

April 20th, 2011

A Pilot Project on the Use of Prediction Markets

in Information Security

alpha-pilot at securitypredictions dot com

Overview   Motivation (dg)

  Prediction Market Examples (gs)

  What is the pilot; what information will it generate? (gs)

  Why is this valuable to the infosec industry? (ah)

  How is this helpful to security teams and professionals? (ah)

April 2011 2 Pilot Project for an InfoSec Prediction Market Geer Hutton Shannon

Motivations   Our Goal: Accelerated aggregation and dissemination of

actionable security information from diverse sources

  Purpose of this talk: Explain the Pilot Project

  Purpose of the pilot: Validate that we can use a market to collect informed opinions from participants that when aggregated and shared is of interest to individuals, organizations and the information security industry.

  Excellent overview and references in:   "Using Prediction Markets to Enhance US Intelligence Capabilities," CIA Center for

the Study of Intelligence, 2006, v50 n6, PDF 17pp. http://tinyurl.com/6kdqpl


The Art in Prediction

  In prediction markets, the art is selecting the questions, i.e., prediction markets are invulnerable to idiots but not to idiotic questions.

  Science and practice alike have shown that prediction markets have greater accuracy than surveys and, unlike surveys, can be run continuously.

  As the rewards available to market participants rise, the precision of the market's predictions improves.


Successful Public Prediction Markets

Primer


A Simple Market Example

  http://en.wikipedia.org/wiki/Prediction_market   Will candidate X win election Y? Yes or no?

  Three elements: Participants, Contracts, Incentives


What are Prediction Markets?

“Large groups of people are smarter than an elite few,

no matter how brilliant — better at solving problems, fostering innovation, coming to wise decisions,

even predicting the future.” — James Surowiecki, author of “The Wisdom of Crowds”

def. Speculative markets used to make predictions of specific events. Contracts representing the event, or outcome, are bought and sold resulting in contract price fluctuations. The current price represents the current group estimate of the likelihood of the event.

Primer

April 2011 7

How They Work: Reflecting Confidence in Outcomes

  Individual answers are anonymous, market aggregates consensus   Participants are incented to express the strength of their confidence   Participants are rewarded based on the accuracy of their contributions   Social collaboration and comments by question, surface root causes

April 2011 8

How They Work: Revealing Early Warning Indicators

  Participants invest in stocks (buy/sell) and thus drive the price up or down. The price reflects the crowd’s confidence in the stated outcome.

  Decision-makers receive an analytical, real-time consensus view into the true state of key issues.

Information contained in dropping confidence

Project Aries will achieve customer acceptance by 30-Sept-2011.

April 2011 9

Social Analytic Reports & Decision Dashboards

Tracking changing trends in consensus opinions

Identifying divergent opinions among participants subgroups – where does the information reside?

Monitor par*cipa*on to ensure diversity

April 2011 10

Pilot Overview   60-day alpha pilot   Use Consensus Point as the market platform   20-30 hand-picked participants   Internal (market) recognition as the incentive   Binary contracts varying in topic and duration

  Written by Geer, Hutton, Shannon   Pilot objectives:

  At least 10 contracts open at all times   20 contracts with at least 10 participants,100 trades   Positive survey results from participants at the end   At least 3 unclosed contracts estimating future events   Have a contract payout on an unexpected security event   Gain enough confidence to start a half-year beta


What Do We Want To Know?   What is the collective, anonymous, incented opinion

about actionable information security events and states of the world?

  How accurate and stable is this opinion/knowledge?

  Can this knowledge benefit participants, 3rd parties and the industry to improve information security?

  Can a prediction market mitigate the unavailability of detailed operational infosec data?


Criteria For Contracts   A binary question

  Good: The market-cap leader in consumer operating systems issues a press-release on a security-critical patch this quarter.

  Poor: The number of software vulnerabilities discovered in the most popular consumer operating system increased this quarter over the previous quarter.

  A definitive authority on the result   Good: government agency, public company, nationally-recognized institution   Poor: news, an individual, on-line poll, micro-blog traffic

  A history of indisputable previous outcomes   Good: Alerts issued, scores published, reports published   Poor: News articles, court documents, non-public sources

  Market information is likely actionable   Good: A disruptive OS patch is in the pipeline   Poor: Companies will lose more data this year than last

  Morally benign

  Difficult for single entities to influence the outcome of the underlying event


Candidate Contracts


Other Candidate Sources & Contracts   US-CERT alerts   Botnet species announced   Statistics from data breach reports   Trends in security surveys and indexes   Statistics from software security or controls reports   MITRE CVE reports


Criteria for Alpha Participants   Demonstrated knowledge of information security   At least 5 years of professional experience in such   Diverse across

  Sectors: Government, Industry, Academic   Verticals: Civilian Gov’t, Health, Financial, DoD, Telecom, etc.   Layers: hosts, networks, applications, infrastructure, content   Life cycle: creation, installation, operation, incidents, remediation   Specialties: privacy, risk, availability, integrity, etc.   Demographics


Incentive Criteria   Is legal

  Is sufficient to entice participants to divulge their knowledge through market activity

  Benefits are tangible to all participants   Not just the top performers

  Does not encourage market manipulation or spectuation

  Scales to 50 active contacts and 1,000 participants


Value to the InfoSec Industry

  Opportunity for big-time benefit to the industry.



  A prediction market is a specifically framed piece of knowledge (belief as a probability)

  What do you want knowledge about?   Understand trends as they happen (or don’t happen)



threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested context: Capability to manage (skills, resources, decision quality…)



  Example: Mobile Malware

  % Mobile devices as targeted asset in 2011 DBIR   % Mobile devices as targeted asset in 2012 DBIR   % Mobile devices as targeted asset in 2013 DBIR

  The effect of new vulnerability research on the above contracts...   The effect of new security technologies on the above contracts...



threat landscape

asset landscape

impact landscape

controls landscape

risk

Suggested context: Capability to manage (skills, resources, decision quality…)


Value to InfoSec Teams and Professionals   An internally facing prediction market can be used for

decision support   Success/Failure of big dollar security projects   What current projects (both security and non-security) mean

to the frequency or impact of security events   Impact of current security events

  This breach will cost how much?


Value to InfoSec Teams and Professionals

  Calibration   Ability to better qualify the subjective evidence around us

  Ability to “mine” changes in “price” for causes


Recap

  Our Goal: Accelerated aggregation and dissemination of actionable security information from diverse sources

  To follow or join the pilot send e-mail to: alpha-pilot at security predictions dot com


On The Use of Prediction Markets in Information Security (from src-bos program)

A tool created to help establish beliefs as probabilities, prediction markets are speculative markets created for the purpose of understand the probability of future events. Not widely used in Information Security, Prediction Markets may have benefits to our industry. Dan Geer, Alex Hutton and Greg Shannon will give a background around what prediction markets are, how they can be used by the information security industry as a whole, and how security departments and professionals can use them as a tool to help defend their environments.

Dan Geer is a computer security analyst and risk management specialist and currently the chief information security officer for In-Q-Tel.

Alex Hutton is a principal for Research & Intelligence with the Verizon Business RISK Team.

Dr. Greg Shannon is the chief scientist for the CERT® Program at Carnegie Mellon University’s Software Engineering Institute.

http://www.sourceconference.com/boston/speakers_2011.asp#dgeer

April 2011 26 Pilot Project for an InfoSec Prediction Market

Geer Hutton Shannon

geer - hutton - shannon - a pilot project on the use of prediction markets in information security

Technology