Evolution, Deception and Terror

Ross Anderson

Cambridge

What’s Dependability?

• We’re building big complex socio-technical systems:– The global card payments system

– The European smart grid

– Facebook

– The NHS ‘database’

– …

• What does it take for these systems to be dependable?

Economics

• What does it even mean for these systems to be dependable?– Payments system – who bears the cost of fraud?– Smart grid – meters report to power company, or

government?– …

• With many players, you need an equilibrium arising out of players’ incentives

• Approaches include security economics, mechanism design, …

Example – Facebook

• Clear conflict of interest– Facebook wants to sell user data– Users want feeling of intimacy, small group, social

control

• Complex access controls – 60+ settings on 7 pages• Privacy almost never salient (why?)• Over 90% of users never change defaults• This lets Facebook blame the customer when

things go wrong

Privacy• Most people say they value privacy, but act

otherwise. Most privacy ventures failed

• Why this privacy gap?

• Odlyzko – technology makes price discrimination both easier and more attractive

• Acquisti – people care about privacy when buying clothes, but not cameras

• Loewenstein – privacy salience. Do stable privacy preferences even exist at all?

Social Engineering

• Use a plausible story, or just bully the target• ‘What’s your PIN so I can cancel your card?’• NYHA case• Patricia Dunn case• Kevin Mitnick ‘Art of Deception’• Traditional responses:

– mandatory access control

– operational security

Social Engineering (2)

• Social psychology:– Solomon Asch, 1951: two-thirds of subjects would

deny obvious facts to conform to group

– Stanley Milgram, 1964: a similar number will administer torture if instructed by an authority figure

– Philip Zimbardo, 1971: you don’t need authority: the subjects’ situation / context is enough

• The Officer Scott case• And what about users you can’t train (customers)?

Usability and Psychology• ‘Why Johnny Can’t Encrypt’ – study of

encryption program PGP – showed that 90% of users couldn’t get it right give 90 minutes

• Private / public, encryption / signing keys, plus trust labels was too much – people would delete private keys, or publish them, or whatever

• Our 1998 study of password advice: mnemonics best, compliance still patchy

• Security is hard – unmotivated users, abstract security policies, lack of feedback …

Phishing

• Started in 2003 with six reported (there had been isolated earlier attacks on AOL passwords)

• By 2006, UK banks lost £35m (£33m by one bank) and US banks maybe $200m

• Early phish crude and greedy but phishermen learned fast

• E.g. ‘Thank you for adding a new email address to your PayPal account’

• The banks make it easy for them – e.g. Halifax

Phishing (2)

• Banks pay firms to take down phishing sites• A couple have moved to two-factor authentication

(CAP) – has its own problems• At present, the phished banks are those with poor

back-end controls and slow asset recovery• One gang (Rockphish) is doing half to two-thirds

of the business• Mule recruitment seems to be a serious bottleneck

Fraud and Phishing Patterns

• Fraudsters do pretty well everything normal marketers do

• The IT industry has abandoned manuals – people learn by doing, and marketers train them in unsafe behaviour (click on links…)

• Banks’ approach is ‘blame and train’ – long known to not work in safety critical systems

• Their instructions ‘look for the lock’, ‘click on images not URLs’, ‘parse the URL’ are easily turned round, and discriminate against nongeeks

Results

• Ability to detect phishing is correlated with SQ-EQ

• It is (independently) correlated with gender

• So the gender HCI issue applies to security too

Marketing Psychology

• See, for example, Cialdini’s “Influence – Science and Practice”

• People make buying decisions with the emotions and rationalise afterwards

• Mostly we’re too busy to research each purchase – and in the ancestral evolutionary environment we had to make flight-or-fight decisions quickly

• The older parts of the brain kept us alive for millions of years before we became sentient

• We still use them more than we care to admit!

Marketing Psychology (2)

• Mental shortcuts include quality = price and quality = scarcity

• Reciprocation can be used to draw people in• Then get a commitment and follow through• Cognitive dissonance: people want to be

consistent (or at least to think that they are)• Social proof: like to do what others do• People also like to defer to authority• They want to deal with people they can relate to

Prospect theory

• Kahneman & Tversky, 1970s: people value gains and losses differently

• Evolutionary logic of risk aversion, status quo bias• Can drive fear marketing, ‘savings’, and (some of the)

irrational behaviour of financial markets

Context and Framing• Framing effects include ‘Was £8.99 now £6.99’

and the estate agent who shows you a crummy house first

• Take along an ugly friend on a double date …• Typical phishing attack: user is fixated on task

completion (e.g. finding why new payee on PayPal account)

• Advance fee frauds take this to extreme lengths!• Risk salience is hugely dependent on context! E.g.

CMU experiment on privacy

Risk Misperception• Terrorist tactics have evolved over centuries to

exploit our mental heuristics and biases• Risk aversion – we are oversensitive to low-

probability, highly-damaging events• Loewnstein & O’Donoghue “Animal Spirits”:

model our objective function by U + h(w)M, where U is rational utility from deliberative system and M is from affective system

• U does Bayesian probability, M just does averages, w is willpower

• Explains other stuff (e.g. hyperbolic discounting)

Risk Misperception (2)

• Loewenstein-O’Donoghue model may give quantitative insight into ‘Availability heuristic’ – easily-recalled data used to frame assessments

• Add: extra credence given to images • Also: our behaviour evolved in small social

groups, and we react against the out-group• We are also sensitive to agency, and in particular

to hostile intentions

Risk Misperception (3)

• Mortality salience greatly amplifies all this• Pyszczynski and colleagues: the experiment with

the Tucson judges• And it’s not just condemnation of the wicked…• Even taking one group past a graveyard is enough

of a ‘memento mori’• So what chance has ‘cyber-terrorism’ got?

So What about Terrorism?

• People learn! – the lesson from auctions; UK/USA• Politicians learn too! Mueller on attitudes of

different US presidents, at the time and later• But what’s next – will it get ever sneakier and

nastier, just as marketing does?• Mueller’s stats; Collier on greed and grievance• Limits on asymmetry? Network effects? What else? • How would a capable green terror group operate?