The Economics and The Economics and Psychology of SecurityPsychology of Security

Ross AndersonRoss Anderson

Cambridge UniversityCambridge University

Social Science and SecuritySocial Science and Security The link between economics and security The link between economics and security

atrophied after WW2atrophied after WW2 Since 2000, we have started to apply economic Since 2000, we have started to apply economic

analysis to IT security and dependabilityanalysis to IT security and dependability Economic analysis often explains failure better Economic analysis often explains failure better

then technical analysis!then technical analysis! Infosec mechanisms are used increasingly to Infosec mechanisms are used increasingly to

support business models (DRM, accessory support business models (DRM, accessory control) rather than to manage riskcontrol) rather than to manage risk

Economic analysis is also vital for the public policy Economic analysis is also vital for the public policy aspects of securityaspects of security

Sociology and psychology are now engaged tooSociology and psychology are now engaged too

Traditional View of InfosecTraditional View of Infosec

People used to think that the Internet was People used to think that the Internet was insecure because of lack of features – insecure because of lack of features – crypto, authentication, filteringcrypto, authentication, filtering

So engineers worked on providing better, So engineers worked on providing better, cheaper security features – AES, PKI, cheaper security features – AES, PKI, firewalls …firewalls …

About 1999, we started to realize that this About 1999, we started to realize that this is not enoughis not enough

Incentives and InfosecIncentives and Infosec

Electronic banking: UK banks were less liable for Electronic banking: UK banks were less liable for fraud, so ended up suffering more internal fraud fraud, so ended up suffering more internal fraud and more errorsand more errors

Distributed denial of service: viruses now don’t Distributed denial of service: viruses now don’t attack the infected machine so much as using it attack the infected machine so much as using it to attack othersto attack others

Health records: hospitals, not patients, buy IT Health records: hospitals, not patients, buy IT systems, so they protect hospitals’ interests systems, so they protect hospitals’ interests rather than patient privacyrather than patient privacy

Why is Microsoft software so insecure, despite Why is Microsoft software so insecure, despite market dominance?market dominance?

New View of InfosecNew View of Infosec

Systems are often insecure because the people Systems are often insecure because the people who guard them, or who could fix them, have who guard them, or who could fix them, have insufficient incentivesinsufficient incentives Bank customers suffer when poorly-designed bank Bank customers suffer when poorly-designed bank

systems make fraud and phishing easiersystems make fraud and phishing easier Patients suffer when hospital systems break privacyPatients suffer when hospital systems break privacy Casino websites suffer when infected PCs run DDoS Casino websites suffer when infected PCs run DDoS

attacks on themattacks on them Insecurity is often what economists call an Insecurity is often what economists call an

‘externality’ – a side-effect, like environmental ‘externality’ – a side-effect, like environmental pollutionpollution

New Uses of InfosecNew Uses of Infosec

Xerox started using authentication in ink Xerox started using authentication in ink cartridges to tie them to the printer – and its cartridges to tie them to the printer – and its competitors soon followedcompetitors soon followed

Motorola then started authenticating mobile Motorola then started authenticating mobile phone batteries to the phonephone batteries to the phone

Carmakers make ‘chipping’ harder, and plan to Carmakers make ‘chipping’ harder, and plan to authenticate major componentsauthenticate major components

DRM: Apple grabs control of music download, DRM: Apple grabs control of music download, MS trying to do the same for HD video contentMS trying to do the same for HD video content

IT Economics (1)IT Economics (1)

The first distinguishing characteristic of many IT The first distinguishing characteristic of many IT product and service markets is network effectsproduct and service markets is network effects

Metcalfe’s law – the value of a network is the Metcalfe’s law – the value of a network is the square of the number of userssquare of the number of users

Real networks – phones, fax, emailReal networks – phones, fax, email Virtual networks – PC architecture versus MAC, Virtual networks – PC architecture versus MAC,

or Symbian versus WinCEor Symbian versus WinCE Network effects tend to lead to dominant firm Network effects tend to lead to dominant firm

markets where the winner takes allmarkets where the winner takes all

IT Economics (2)IT Economics (2)

Second common feature of IT product and Second common feature of IT product and service markets is high fixed costs and low service markets is high fixed costs and low marginal costsmarginal costs

Competition can drive down prices to marginal Competition can drive down prices to marginal cost of productioncost of production

This can make it hard to recover capital This can make it hard to recover capital investment, unless stopped by patent, brand, investment, unless stopped by patent, brand, compatibility …compatibility …

These effects can also lead to dominant-firm These effects can also lead to dominant-firm market structuresmarket structures

IT Economics (3)IT Economics (3)

Third common feature of IT markets is that Third common feature of IT markets is that switching from one product or service to another switching from one product or service to another is expensiveis expensive

E.g. switching from Windows to Linux means E.g. switching from Windows to Linux means retraining staff, rewriting appsretraining staff, rewriting apps

Shapiro-Varian theorem: the net present value of Shapiro-Varian theorem: the net present value of a software company is the total switching costsa software company is the total switching costs

This is why so much effort goes into managing This is why so much effort goes into managing switching costs – once you have $3000 worth of switching costs – once you have $3000 worth of songs on a $300 iPod, you’re locked into iPodssongs on a $300 iPod, you’re locked into iPods

IT Economics and SecurityIT Economics and Security

High fixed/low marginal costs, network effects High fixed/low marginal costs, network effects and switching costs all tend to lead to dominant-and switching costs all tend to lead to dominant-firm markets with big first-mover advantagefirm markets with big first-mover advantage

So time-to-market is criticalSo time-to-market is critical Microsoft philosophy of ‘we’ll ship it Tuesday and Microsoft philosophy of ‘we’ll ship it Tuesday and

get it right by version 3’ is not perverse get it right by version 3’ is not perverse behaviour by Bill Gates but quite rationalbehaviour by Bill Gates but quite rational

Whichever company had won in the PC OS Whichever company had won in the PC OS business would have done the samebusiness would have done the same

IT Economics and Security (2)IT Economics and Security (2)

When building a network monopoly, you must When building a network monopoly, you must appeal to vendors of complementary productsappeal to vendors of complementary products

That’s application software developers in the That’s application software developers in the case of PC versus Apple, or now of Symbian case of PC versus Apple, or now of Symbian versus WinCE, or WinMP versus Realversus WinCE, or WinMP versus Real

Lack of security in earlier versions of Windows Lack of security in earlier versions of Windows made it easier to develop applicationsmade it easier to develop applications

So did the choice of security technologies that So did the choice of security technologies that dump most costs on the user (SSL, PKI, …)dump most costs on the user (SSL, PKI, …)

Once you’re a monopolist, lock it all down!Once you’re a monopolist, lock it all down!

Why are so many security Why are so many security products ineffective?products ineffective?

Akerlof’s Nobel-prizewinning paper, ‘The Market Akerlof’s Nobel-prizewinning paper, ‘The Market for Lemons’ introduced asymmetric informationfor Lemons’ introduced asymmetric information

Suppose a town has 100 used cars for sale: 50 Suppose a town has 100 used cars for sale: 50 good ones worth $2000 and 50 lemons worth good ones worth $2000 and 50 lemons worth $1000$1000

What is the equilibrium price of used cars in this What is the equilibrium price of used cars in this town?town?

If $1500, no good cars will be offered for sale …If $1500, no good cars will be offered for sale …

Security and LiabilitySecurity and Liability

Why did digital signatures not take off?Why did digital signatures not take off? Industry thought: legal uncertainty. So EU Industry thought: legal uncertainty. So EU

passed electronic signature lawpassed electronic signature law But: customers and merchants resist But: customers and merchants resist

transfer of liability by bankers for disputed transfer of liability by bankers for disputed transactionstransactions

If you’re a customer, best stick with credit If you’re a customer, best stick with credit cards, so fraud remains largely the bank’s cards, so fraud remains largely the bank’s problemproblem

PrivacyPrivacy Most people say they value privacy, but act Most people say they value privacy, but act

otherwise. Most privacy technology firms failedotherwise. Most privacy technology firms failed Acquisti – people care about privacy when Acquisti – people care about privacy when

buying clothes, but not cameras (data relating to buying clothes, but not cameras (data relating to body or image are more privacy sensitive)body or image are more privacy sensitive)

Issue for mobile phone industry – phone viruses Issue for mobile phone industry – phone viruses worse for image than PC virusesworse for image than PC viruses

Varian – you can maybe fix privacy by giving Varian – you can maybe fix privacy by giving people property rights in personal informationpeople property rights in personal information

Odlyzko – technology makes price discrimination Odlyzko – technology makes price discrimination both easier and more attractiveboth easier and more attractive

Why Bill wasn’t interested in Why Bill wasn’t interested in securitysecurity

While Microsoft was growing, the two While Microsoft was growing, the two critical factors were speed, and appeal to critical factors were speed, and appeal to application developersapplication developers

Security markets were over-hyped and Security markets were over-hyped and driven by artificial factorsdriven by artificial factors

Issues like privacy and liability were more Issues like privacy and liability were more complex than they seemedcomplex than they seemed

The public couldn’t tell good security from The public couldn’t tell good security from bad anywaybad anyway

Why is Bill now changing his Why is Bill now changing his mind?mind?

Security can help lock customers in, and extend Security can help lock customers in, and extend power from one market to anotherpower from one market to another

Information Rights Management changes Information Rights Management changes ownership of a file from the machine owner to ownership of a file from the machine owner to the file creatorthe file creator

Remember: value of software company = total Remember: value of software company = total switching costs. And once documents can’t be switching costs. And once documents can’t be converted without creators’ permission, the converted without creators’ permission, the switching cost is much higherswitching cost is much higher

And: will WMP/Vista let Microsoft do to high And: will WMP/Vista let Microsoft do to high definition movies what Apple did for music?definition movies what Apple did for music?

Open versus Closed?Open versus Closed?

Are open-source systems more dependable? It’s Are open-source systems more dependable? It’s easier for the attackers to find vulnerabilities, but easier for the attackers to find vulnerabilities, but also easier for the defenders to find and fix themalso easier for the defenders to find and fix them

Theory: openness helps both equally if bugs are Theory: openness helps both equally if bugs are random and standard dependability model random and standard dependability model assumptions applyassumptions apply

Statistics: bugs are correlated in a number of Statistics: bugs are correlated in a number of real systems (‘Milk or Wine?’)real systems (‘Milk or Wine?’)

Trade-off: the gains from this, versus the risks to Trade-off: the gains from this, versus the risks to systems whose owners don’t patchsystems whose owners don’t patch

How Much to Spend?How Much to Spend?

How much should the average company How much should the average company spend on information security?spend on information security?

Governments, vendors say: much much Governments, vendors say: much much more than at present!more than at present!

But they’ve been saying this for 20 years!But they’ve been saying this for 20 years! Measurements of security return-on-Measurements of security return-on-

investment suggest about 20% p.a. overallinvestment suggest about 20% p.a. overall So the total expenditure may be about So the total expenditure may be about

rightright

Skewed IncentivesSkewed Incentives

Why do large companies spend too much on Why do large companies spend too much on security and small companies too little?security and small companies too little?

Research shows there’s an adverse selection Research shows there’s an adverse selection effecteffect

Corporate security managers tend to be risk-Corporate security managers tend to be risk-averse people, often from accounting / financeaverse people, often from accounting / finance

More risk-loving people may become sales or More risk-loving people may become sales or engineering staff, or small-firm entrepreneursengineering staff, or small-firm entrepreneurs

There’s also due-diligence, government There’s also due-diligence, government regulation, and insurance to think ofregulation, and insurance to think of

Skewed Incentives (2)Skewed Incentives (2)

If you are DirNSA and have a nice new If you are DirNSA and have a nice new hack on XP and Vista, do you tell Bill?hack on XP and Vista, do you tell Bill?

Tell – protect 300m AmericansTell – protect 300m Americans Don’t tell – be able to hack 400m Don’t tell – be able to hack 400m

Europeans, 1000m Chinese,…Europeans, 1000m Chinese,… If the Chinese hack US systems, they If the Chinese hack US systems, they

keep quiet. If you hack their systems, you keep quiet. If you hack their systems, you can brag about it to the Presidentcan brag about it to the President

So offence can be favoured over defenceSo offence can be favoured over defence

Large Project FailureLarge Project Failure

Maybe 30% of large projects fail Maybe 30% of large projects fail But we build much bigger failures But we build much bigger failures

nowadays than 30 years ago so…nowadays than 30 years ago so… Why do more public-sector projects fail?Why do more public-sector projects fail? Consider what the incentives are on Consider what the incentives are on

project managers versus ministers – and project managers versus ministers – and what sort of people will become successful what sort of people will become successful project managers versus ministers!project managers versus ministers!

Security and SociologySecurity and Sociology

There’s a lot of interest recently in using social There’s a lot of interest recently in using social networks to analyse interactions and systemsnetworks to analyse interactions and systems

Barabási and Albert showed that a scale-free Barabási and Albert showed that a scale-free network could be attacked efficiently by targeting network could be attacked efficiently by targeting its high-order nodesits high-order nodes

Think: rulers target Saxon landlords / Ukrainian Think: rulers target Saxon landlords / Ukrainian kulaks / Tutsi schoolteachers /…kulaks / Tutsi schoolteachers /…

Can we use evolutionary game theory ideas to Can we use evolutionary game theory ideas to figure out how networks evolve?figure out how networks evolve?

Idea: run many simulations between different Idea: run many simulations between different attack / defence strategiesattack / defence strategies

Security and Sociology (2)Security and Sociology (2)

Vertex-order attacks with:Vertex-order attacks with: Black – normal (scale-Black – normal (scale-

free) node free) node replenishmentreplenishment

Green – defenders Green – defenders replace high-order replace high-order nodes with ringsnodes with rings

Cyan – they use Cyan – they use cliques (c.f. system cliques (c.f. system biology …)biology …)

Psychology and SecurityPsychology and Security

Fastest growing online crime is phishing – it only Fastest growing online crime is phishing – it only started in 2004, but by 2006 it cost the UK £35m started in 2004, but by 2006 it cost the UK £35m and the USA perhaps $200mand the USA perhaps $200m

‘‘Pretexting’ always existed (see Mitnick’s book), Pretexting’ always existed (see Mitnick’s book), but phishing industrializes itbut phishing industrializes it

In a company you can train the staff in In a company you can train the staff in operational security (though many don’t). It’s operational security (though many don’t). It’s harder when the target is your users!harder when the target is your users!

Maybe more secure machines would inevitably Maybe more secure machines would inevitably drive the bad guys to target the people insteaddrive the bad guys to target the people instead

What can security folks learn from psychology?What can security folks learn from psychology?

Psychology and Security (2)Psychology and Security (2)

Security usability research is fairly new and the Security usability research is fairly new and the results are pessimistic: most security products results are pessimistic: most security products don’t work well or at alldon’t work well or at all

Over half of all SSL certificates are ‘wrong’Over half of all SSL certificates are ‘wrong’ No problem – we train people to keep on clicking No problem – we train people to keep on clicking

‘OK’ until they can get their work done‘OK’ until they can get their work done Banks react to phishing by ‘blame and train’ Banks react to phishing by ‘blame and train’

efforts towards customers – but we know from efforts towards customers – but we know from the safety-critical world that this doesn’t workthe safety-critical world that this doesn’t work

Systems designed by geeks discriminate against Systems designed by geeks discriminate against women, the elderly and the less educatedwomen, the elderly and the less educated

Psychology and Security (3)Psychology and Security (3)

Social psychology has long been relevant to us!Social psychology has long been relevant to us! Solomon Asch showed most people would deny the Solomon Asch showed most people would deny the

evidence of their eyes to conform to a groupevidence of their eyes to conform to a group Stanley Milgram showed that 60% of people will do Stanley Milgram showed that 60% of people will do

downright immoral things if ordered todownright immoral things if ordered to Philip Zimbardo’s Stanford Prisoner Experiment Philip Zimbardo’s Stanford Prisoner Experiment

showed roles and group dynamics were enoughshowed roles and group dynamics were enough The disturbing case of ‘Officer Scott’The disturbing case of ‘Officer Scott’ How can systems resist abuse of authority?How can systems resist abuse of authority? Why do people need enemies?Why do people need enemies? Why does terrorism work?Why does terrorism work?

Psychology and Security (4)Psychology and Security (4)

Evolutionary psychology may eventually explain Evolutionary psychology may eventually explain cognitive biases. It is based on the massive cognitive biases. It is based on the massive modularity hypothesis and the use of FMRI to modularity hypothesis and the use of FMRI to track brain functiontrack brain function

Simon Baron-Cohen’s work on autism suggests Simon Baron-Cohen’s work on autism suggests a ‘theory of mind’ module central to empathy for a ‘theory of mind’ module central to empathy for others’ mental statesothers’ mental states

This is how we differ from the great apesThis is how we differ from the great apes It helps us lie, and to detect lies told by others It helps us lie, and to detect lies told by others So are we really homo sapiens sapiens – or So are we really homo sapiens sapiens – or

homo sapiens deceptor?homo sapiens deceptor?

The Information SocietyThe Information Society

More and more goods contain softwareMore and more goods contain software More and more industries are starting to More and more industries are starting to

become like the software industrybecome like the software industry The good: flexibility, rapid responseThe good: flexibility, rapid response The bad: frustration, poor serviceThe bad: frustration, poor service The ugly: monopoliesThe ugly: monopolies How will society evolve to cope?How will society evolve to cope?

The Research AgendaThe Research Agenda

We need to figure out how to balance competing We need to figure out how to balance competing social goals, as we have in the physical worldsocial goals, as we have in the physical world

Security economics gives us tools to understand Security economics gives us tools to understand what’s going on and to analyse policy optionswhat’s going on and to analyse policy options

Sociology also gives some useful insightsSociology also gives some useful insights And security psychology is not just a side And security psychology is not just a side

discipline relevant to usability and phishing – it discipline relevant to usability and phishing – it has the potential to bring us fundamental has the potential to bring us fundamental insights, just as security economics hasinsights, just as security economics has

More …More …

Economics and Security Resource Page – Economics and Security Resource Page – www.www.clcl.cam.ac..cam.ac.ukuk/~rja14//~rja14/econsececonsec.html.html (or (or follow link from follow link from www.www.rossross--andersonanderson.com.com) )

WEIS – Annual Workshop on Economics WEIS – Annual Workshop on Economics and Information Security – next at CMU, and Information Security – next at CMU, June 7–8 2006June 7–8 2006

Foundation for Information Policy Foundation for Information Policy Research – Research – www.www.fiprfipr.org.org