towards a science of security and human behaviour ross anderson cambridge university

Towards a Science of Towards a Science of Security and Human Security and Human

BehaviourBehaviour

Ross AndersonRoss Anderson

Cambridge UniversityCambridge University

SOUPS 2008 July 24th 2008

Traditional View of InfosecTraditional View of Infosec

People used to think that the Internet was People used to think that the Internet was insecure because of lack of features – insecure because of lack of features – crypto, authentication, filteringcrypto, authentication, filtering

So we all worked on providing better, So we all worked on providing better, cheaper security features – AES, PKI, cheaper security features – AES, PKI, firewalls …firewalls …

About 1999, some of us started to realize About 1999, some of us started to realize that this is not enoughthat this is not enough


Economics and SecurityEconomics and Security Since 2000, we have started to apply economic Since 2000, we have started to apply economic

analysis to IT security and dependabilityanalysis to IT security and dependability It often explains failure better! It often explains failure better! Electronic banking: UK banks were less liable for Electronic banking: UK banks were less liable for

fraud, so ended up suffering more internal fraud fraud, so ended up suffering more internal fraud and more errorsand more errors

Distributed denial of service: viruses now don’t Distributed denial of service: viruses now don’t attack the infected machine so much as using it attack the infected machine so much as using it to attack othersto attack others

Why is Microsoft software so insecure, despite Why is Microsoft software so insecure, despite market dominance?market dominance?


New View of InfosecNew View of Infosec

Systems are often insecure because the Systems are often insecure because the people who guard them, or who could fix people who guard them, or who could fix them, have insufficient incentivesthem, have insufficient incentives Bank customers suffer when poorly-designed Bank customers suffer when poorly-designed

bank systems make fraud and phishing easierbank systems make fraud and phishing easier Casino websites suffer when infected PCs run Casino websites suffer when infected PCs run

DDoS attacks on themDDoS attacks on them Insecurity is often what economists call an Insecurity is often what economists call an

‘externality’ – a side-effect, like ‘externality’ – a side-effect, like environmental pollutionenvironmental pollution


New Uses of InfosecNew Uses of Infosec

Xerox started using authentication in ink Xerox started using authentication in ink cartridges to tie them to the printer – and cartridges to tie them to the printer – and its competitors soon followedits competitors soon followed

Carmakers make ‘chipping’ harder, and Carmakers make ‘chipping’ harder, and plan to authenticate major componentsplan to authenticate major components

DRM: Apple grabs control of music DRM: Apple grabs control of music download, MS accused of making a play download, MS accused of making a play to control distribution of HD video contentto control distribution of HD video content


IT Economics (1)IT Economics (1)

The first distinguishing characteristic of many IT The first distinguishing characteristic of many IT product and service markets is network effectsproduct and service markets is network effects

Metcalfe’s law – the value of a network is the Metcalfe’s law – the value of a network is the square of the number of userssquare of the number of users

Real networks – phones, fax, emailReal networks – phones, fax, email Virtual networks – PC architecture versus MAC, Virtual networks – PC architecture versus MAC,

or Symbian versus WinCEor Symbian versus WinCE Network effects tend to lead to dominant firm Network effects tend to lead to dominant firm

markets where the winner takes allmarkets where the winner takes all



Second common feature of IT product and Second common feature of IT product and service markets is high fixed costs and low service markets is high fixed costs and low marginal costsmarginal costs

Competition can drive down prices to marginal Competition can drive down prices to marginal cost of productioncost of production

This can make it hard to recover capital This can make it hard to recover capital investment, unless stopped by patent, brand, investment, unless stopped by patent, brand, compatibility …compatibility …

These effects can also lead to dominant-firm These effects can also lead to dominant-firm market structuresmarket structures



Third common feature of IT markets is that Third common feature of IT markets is that switching from one product or service to another switching from one product or service to another is expensiveis expensive

E.g. switching from Windows to Linux means E.g. switching from Windows to Linux means retraining staff, rewriting appsretraining staff, rewriting apps

Shapiro-Varian theorem: the net present value of Shapiro-Varian theorem: the net present value of a software company is the total switching costsa software company is the total switching costs

So major effort goes into managing switching So major effort goes into managing switching costs – once you have $3000 worth of songs on costs – once you have $3000 worth of songs on a $300 iPod, you’re locked into iPodsa $300 iPod, you’re locked into iPods


IT Economics and SecurityIT Economics and Security

High fixed/low marginal costs, network effects High fixed/low marginal costs, network effects and switching costs all tend to lead to dominant-and switching costs all tend to lead to dominant-firm markets with big first-mover advantagefirm markets with big first-mover advantage

So time-to-market is criticalSo time-to-market is critical Microsoft philosophy of ‘we’ll ship it Tuesday and Microsoft philosophy of ‘we’ll ship it Tuesday and

get it right by version 3’ is not perverse get it right by version 3’ is not perverse behaviour by Bill Gates but quite rationalbehaviour by Bill Gates but quite rational

Whichever company had won in the PC OS Whichever company had won in the PC OS business would have done the samebusiness would have done the same


IT Economics and Security (2)IT Economics and Security (2)

When building a network monopoly, you must When building a network monopoly, you must appeal to vendors of complementary productsappeal to vendors of complementary products

That’s application software developers in the That’s application software developers in the case of PC versus Apple, or now of Symbian case of PC versus Apple, or now of Symbian versus Linux/Windows/J2EE/Palmversus Linux/Windows/J2EE/Palm

Lack of security in earlier versions of Windows Lack of security in earlier versions of Windows made it easier to develop applicationsmade it easier to develop applications

So did the choice of security technologies that So did the choice of security technologies that dump usability costs on the user (SSL, not SET)dump usability costs on the user (SSL, not SET)

Once you’ve a monopoly, lock it all down!Once you’ve a monopoly, lock it all down!


Economics and UsabilityEconomics and Usability

Make your products usable by newbiesMake your products usable by newbies … … but much more usable with practice!but much more usable with practice! To what extent can you make skill a To what extent can you make skill a

source of asymmetric lockin?source of asymmetric lockin? Hypothesis: this underlies the failure of Hypothesis: this underlies the failure of

user programmability to get traction!user programmability to get traction! We have nothing now as good as BASIC We have nothing now as good as BASIC

was in the 1980s…was in the 1980s…


Economics and Usability (2)Economics and Usability (2)

How many features should my product have?How many features should my product have? Marginal benefit of new feature concentrated in Marginal benefit of new feature concentrated in

some target marketsome target market Marginal cost spread over all usersMarginal cost spread over all users So we get chronic featuritis!So we get chronic featuritis! At equilibrium, a computer / phone / anything At equilibrium, a computer / phone / anything

programmable will be just on the edge of programmable will be just on the edge of unacceptability to a significant number of usersunacceptability to a significant number of users

The same happens with laws, services, …The same happens with laws, services, …


Why are so many security Why are so many security products ineffective?products ineffective?

Akerlof’s Nobel-prizewinning paper, ‘The Market Akerlof’s Nobel-prizewinning paper, ‘The Market for Lemons’ introduced asymmetric informationfor Lemons’ introduced asymmetric information

Suppose a town has 100 used cars for sale: 50 Suppose a town has 100 used cars for sale: 50 good ones worth $2000 and 50 lemons worth good ones worth $2000 and 50 lemons worth $1000$1000

What is the equilibrium price of used cars?What is the equilibrium price of used cars? If $1500, no good cars will be offered for sale …If $1500, no good cars will be offered for sale … Started the study of asymmetric informationStarted the study of asymmetric information Security products are often a ‘lemons market’Security products are often a ‘lemons market’


Products worse then uselessProducts worse then useless

Adverse selection and moral hazard matter (why Adverse selection and moral hazard matter (why do Volvo drivers have more accidents?)do Volvo drivers have more accidents?)

Application to trust: Ben Edelman, ‘Adverse Application to trust: Ben Edelman, ‘Adverse selection on online trust certifications’ (WEIS 06)selection on online trust certifications’ (WEIS 06)

Websites with a TRUSTe certification are more Websites with a TRUSTe certification are more than twice as likely to be maliciousthan twice as likely to be malicious

The top Google ad is about twice as likely as the The top Google ad is about twice as likely as the top free search result to be malicious (other top free search result to be malicious (other search engines worse …)search engines worse …)

Conclusion: ‘Don’t click on ads’Conclusion: ‘Don’t click on ads’


PrivacyPrivacy Most people say they value privacy, but act Most people say they value privacy, but act

otherwise. Most privacy ventures failedotherwise. Most privacy ventures failed Why is there this ‘privacy gap’?Why is there this ‘privacy gap’? Odlyzko – technology makes price discrimination Odlyzko – technology makes price discrimination

both easier and more attractiveboth easier and more attractive Acquisti et al – people care about privacy when Acquisti et al – people care about privacy when

buying clothes, but not cameras (phone viruses buying clothes, but not cameras (phone viruses worse for vendor than PC viruses?)worse for vendor than PC viruses?)

Loewenstein et al – it’s not clear that there are Loewenstein et al – it’s not clear that there are stable and coherent privacy preferences! stable and coherent privacy preferences! Student disclosure more for ‘How bad RU’ and Student disclosure more for ‘How bad RU’ and less with detailed privacy noticeless with detailed privacy notice


Conflict theoryConflict theory

Does the defence of a country or a system Does the defence of a country or a system depend on the least effort, on the best effort, or depend on the least effort, on the best effort, or on the sum of efforts?on the sum of efforts?

The last is optimal; the first is really awfulThe last is optimal; the first is really awful Software is a mix: it depends on the worst effort Software is a mix: it depends on the worst effort

of the least careful programmer, the best effort of the least careful programmer, the best effort of the security architect, and the sum of efforts of of the security architect, and the sum of efforts of the testersthe testers

Moral: hire fewer better programmers, more Moral: hire fewer better programmers, more testers, top architectstesters, top architects


How Much to Spend?How Much to Spend?

How much should the average company How much should the average company spend on information security?spend on information security?

Governments, vendors say: much much Governments, vendors say: much much more than at presentmore than at present

But they’ve been saying this for 20 years!But they’ve been saying this for 20 years! Measurements of security return-on-Measurements of security return-on-

investment suggest about 20% p.a. overallinvestment suggest about 20% p.a. overall So the total expenditure may be about So the total expenditure may be about

right. Are there any better metrics?right. Are there any better metrics?


Skewed IncentivesSkewed Incentives

Why do large companies spend too much on Why do large companies spend too much on security and small companies too little?security and small companies too little?

Research shows an adverse selection effectResearch shows an adverse selection effect Corporate security managers tend to be risk-Corporate security managers tend to be risk-

averse people, often from accounting / financeaverse people, often from accounting / finance More risk-loving people may become sales or More risk-loving people may become sales or

engineering staff, or small-firm entrepreneursengineering staff, or small-firm entrepreneurs There’s also due-diligence, government There’s also due-diligence, government

regulation, insurance and agency to think ofregulation, insurance and agency to think of


Skewed Incentives (2)Skewed Incentives (2)

If you are DirNSA and have a nice new If you are DirNSA and have a nice new hack on XP and Vista, do you tell Bill?hack on XP and Vista, do you tell Bill?

Tell – protect 300m AmericansTell – protect 300m Americans Don’t tell – be able to hack 400m Don’t tell – be able to hack 400m

Europeans, 1000m Chinese,…Europeans, 1000m Chinese,… If the Chinese hack US systems, they If the Chinese hack US systems, they

keep quiet. If you hack their systems, you keep quiet. If you hack their systems, you can brag about it to the Presidentcan brag about it to the President

So offence can be favoured over defenceSo offence can be favoured over defence


Security and PolicySecurity and Policy

Our ENISA report, published in March, has Our ENISA report, published in March, has 15 recommendations:15 recommendations: Security breach disclosure lawSecurity breach disclosure law EU-wide data on financial fraudEU-wide data on financial fraud Data on which ISPs host malwareData on which ISPs host malware Slow-takedown penalties and putback rightsSlow-takedown penalties and putback rights Networked devices to be secure by defaultNetworked devices to be secure by default ……

See links from my web pageSee links from my web page


Security and SociologySecurity and Sociology

There’s a lot of interest in using social network There’s a lot of interest in using social network models to analyse systemsmodels to analyse systems

Barabási and Albert showed that a scale-free Barabási and Albert showed that a scale-free network could be attacked efficiently by targeting network could be attacked efficiently by targeting its high-order nodesits high-order nodes

Think: rulers target Saxon landlords / Ukrainian Think: rulers target Saxon landlords / Ukrainian kulaks / Tutsi schoolteachers /…kulaks / Tutsi schoolteachers /…

Can we use evolutionary game theory ideas to Can we use evolutionary game theory ideas to figure out how networks evolve?figure out how networks evolve?

Idea: run many simulations between different Idea: run many simulations between different attack / defence strategiesattack / defence strategies


Security and Sociology (2)Security and Sociology (2)

Vertex-order attacks with:Vertex-order attacks with: Black – normal (scale-Black – normal (scale-

free) replenishmentfree) replenishment Green – defenders Green – defenders

replace high-order replace high-order nodes with ringsnodes with rings

Cyan – they use Cyan – they use cliques (c.f. system cliques (c.f. system biology …)biology …)

Application: traffic Application: traffic analysis (see my analysis (see my Google tech talk)Google tech talk)


Psychology and SecurityPsychology and Security

Phishing only started in 2004, but in 2006 it cost Phishing only started in 2004, but in 2006 it cost the UK £35m and the USA perhaps $200mthe UK £35m and the USA perhaps $200m

Banks react to phishing by ‘blame and train’ Banks react to phishing by ‘blame and train’ efforts towards customersefforts towards customers

But we know from the safety-critical world that But we know from the safety-critical world that this doesn’t work!this doesn’t work!

We train people to keep on clicking ‘OK’ until We train people to keep on clicking ‘OK’ until they can get their work done – and ‘learned they can get their work done – and ‘learned helplessness’ goes much wider helplessness’ goes much wider

People don’t notice missing padlock – the ‘dog People don’t notice missing padlock – the ‘dog that didn’t bark’. Is there anything we can do?that didn’t bark’. Is there anything we can do?


Psychology and Security (2)Psychology and Security (2)

Folklore: systems designed by geeks for Folklore: systems designed by geeks for geeks also discriminate against women, geeks also discriminate against women, the elderly and the less educatedthe elderly and the less educated

We set out to check whether people with We set out to check whether people with higher ‘systemizing’ than ‘empathizing’ higher ‘systemizing’ than ‘empathizing’ ability would detect phishing more easilyability would detect phishing more easily

Methodology: tested students for phishing Methodology: tested students for phishing detection, and also on Baron-Cohen testdetection, and also on Baron-Cohen test

Presented at SHB07: re-examined by sexPresented at SHB07: re-examined by sex


ResultsResults

Ability to detect Ability to detect phishing is correlated phishing is correlated with SQ-EQwith SQ-EQ

It is (independently) It is (independently) correlated with correlated with gendergender

Folklore is right – the Folklore is right – the gender HCI issue gender HCI issue applies to security tooapplies to security too



Social psychology has long been relevant to us!Social psychology has long been relevant to us! Solomon Asch showed most people would deny the Solomon Asch showed most people would deny the

evidence of their eyes to conform to a groupevidence of their eyes to conform to a group Stanley Milgram showed that 60% of people will do Stanley Milgram showed that 60% of people will do

downright immoral things if ordered todownright immoral things if ordered to Philip Zimbardo’s Stanford Prisoner Experiment Philip Zimbardo’s Stanford Prisoner Experiment

showed roles and group dynamics were enoughshowed roles and group dynamics were enough The disturbing case of ‘Officer Scott’The disturbing case of ‘Officer Scott’ How can systems resist abuse of authority?How can systems resist abuse of authority?



Why does terrorism work?Why does terrorism work? The bad news: it’s evolved to exploit a large The bad news: it’s evolved to exploit a large

number of our heuristics and biases!number of our heuristics and biases! Availability heuristic; mortality salience; Availability heuristic; mortality salience;

anchoring; loss aversion in uncertainty; wariness anchoring; loss aversion in uncertainty; wariness of hostile intent; violation of moral sentiments; of hostile intent; violation of moral sentiments; credence given to images; reaction against out-credence given to images; reaction against out-group; sensitivity to change;…group; sensitivity to change;…

The good news: biases affect novel events The good news: biases affect novel events more, and so can be largely overcome by more, and so can be largely overcome by experienceexperience



Deception – from its role in evolution, to Deception – from its role in evolution, to everyday social poker; self-deception; how everyday social poker; self-deception; how deception is different online, and policy… deception is different online, and policy…

Would you really vote for a president you Would you really vote for a president you didn’t think could lie to you?didn’t think could lie to you?

Many inappropriate psychological Many inappropriate psychological ‘interfaces’ are sustained by money or ‘interfaces’ are sustained by money or power – compare why we fear computer power – compare why we fear computer crime too little, and terrorism too muchcrime too little, and terrorism too much


The Research AgendaThe Research Agenda

The online world and the physical world are The online world and the physical world are merging, and this will cause major dislocation for merging, and this will cause major dislocation for many yearsmany years

Security economics gives us some of the tools Security economics gives us some of the tools we need to understand what’s going onwe need to understand what’s going on

Sociology gives some cool and useful stuff tooSociology gives some cool and useful stuff too And security psychology is not just usability and And security psychology is not just usability and

phishing – it might bring us fundamental insights, phishing – it might bring us fundamental insights, just as security economics hasjust as security economics has


More …More …

See See www.www.rossross--andersonanderson.com.com for a survey for a survey article, our ENISA report, my security economics article, our ENISA report, my security economics resource page, and links to:resource page, and links to:

WEIS – Annual Workshop on Economics and WEIS – Annual Workshop on Economics and Information SecurityInformation Security

SHB – Workshop on Security and Human SHB – Workshop on Security and Human Behaviour (www.lightbluetouchpaper.org) Behaviour (www.lightbluetouchpaper.org)

‘‘Security Engineering – A Guide to Building Security Engineering – A Guide to Building Dependable Distributed Systems’ 2e – just out!Dependable Distributed Systems’ 2e – just out!

http://www.ross-anderson.com/





towards a science of security and human behaviour ross anderson cambridge university

Documents

internal fraud

cheaper security features

science of security

designed bank systems

environmental pollution

infected pcs

infected machine

filtering people