Roger G. Johnston, Ph.D., CPP

Right Brain Sekurity

http://rbsekurity.com

+1-630-551-0740

Vulnerability Assessments:

Are You REALLY Doing Them?

Threat: Who might attack, why, when, how,

with what probability, and with what resources.

(Includes information on goals and attack

modes.)

Threat Assessment (TA): Attempting to

identify threats.

Terminology

Vulnerability: Flaw or weakness that could be exploited to cause

undesirable consequences.

Vulnerability Assessment (VA): Creatively devising & discover-

ing (and perhaps demonstrating) ways to defeat a security

device, system, or program. Should include thinking like the bad

guys, and also suggesting countermeasures and security

improvements.

Terminology

mimics what

the bad guys do!

Threat: Adversaries might try to steal PII

information (SSNs, credit card numbers, etc.)

from our computer systems to commit crimes.

Vulnerability: We don’t keep our anti-malware

software up to date.

Threat vs. Vulnerability

4

•Pass a test

•“Test” security

•Generate metrics

•Justify the status quo

•Praise or accuse anybody

•Check against some standard

•Claim there are no vulnerabilities

•Engender warm & happy feelings

•Determine who gets salary increases

•Rationalize the research & development

•Apply a mindless, bureaucratic stamp of approval

•Endorse a security product/program or

Certify it as “good” or “ready to use”

The purpose of a VA is to improve security &

minimize risk, NOT to:

Purpose

5

A VA is Not… pen testing

“Red Teaming”

feature analysis

security auditing

quality control

threat assessment

reliability testing

efficiency testing

software scanning

compliance testing

acceptance testing

ergonomics testing

performance testing

response time testing6

operational assessment

fault or event tree analysis

(from safety engineering)

Design Basis Threat

a security survey

gap analysis

7

Questions Vulnerability Assessors Ask

And You Should, Too

Are vulnerabilities being confused with threats, assets

needing protection, security or infrastructure features, or

attack scenarios?

Are vulnerabilities being thought of as good news?

(They should be!)

Are VAs being confused with other things like TAs or

security “testing”?

Are they being done continuously, or at least frequently?

Vulnerability Assessments

8

Vulnerability Assessments (con’t)

Are the following kinds of employees (even if not security or

cyber experts) drafted to help examine your security:

trouble-makers, creative types, loophole finders,

questioners of authority, skeptics/cynics, hackers,

narcissists, hands-on enthusiasts, and puzzle solvers.

Resiliency & PR preparation for when

security inevitably fails?

Questions Vulnerability Assessors Ask And You Should, Too

9

Vulnerability Assessments (con’t)

Do your VAs suffer from any of these problems?

- sham rigor

- the Fallacy of Precision

- lack of imagination

- reactive not proactive

- done only be insiders

- shooting-the-messenger

- conflicts of interest

- cognitive dissonance

- focused only on high-tech attacks

- artificial constraints (scope, time, effort,

modules/components/disciplines)

- letting the good guys and the current security

infrastructure/strategy define the vulnerabilities &

attacks

Questions Vulnerability Assessors Ask And You Should, Too

10

Questions Vulnerability Assessors Ask And You Should, Too

Use hardware passwords & device IDs?

Have you changed the default password &

device ID, and security settings?

Devices adhere to emerging security standards?

Do the devices follow Minimalist Principles? range & power

duty cycle

bandwidth

data acquisition

data retention & duration

IoT Devices

11

Questions Vulnerability Assessors Ask And You Should, Too

Trusted manufacturers & vendors?

Is security built in from the start, or just a last

minute afterthought?

Early & iterative VAs on the devices?

Secure chain of custody?

IoT Devices (con’t)

12

Questions Vulnerability Assessors Ask And You Should, Too

Are your devices safe from physical/electronic tampering

(~20 secs), counterfeiting, and backdoor insertion including

• at vendor or factory?

• during shipments?

• on loading dock?

• before installation?

• after installation?

Chain of Custody for Devices*

13

Questions Vulnerability Assessors Ask And You Should, Too

Is there a lot of empty space inside your devices? Are they

frequently opened up and examined for tampering and

alien electronics? Do you know what the insides are

supposed to look like? Can you spot a counterfeit device?

Are you under the mistaken impression that:- “anti-counterfeiting” tags (even if high-tech) are difficult to lift or

counterfeit?

- tamper-indicating seals or packaging (even if high-tech) are difficult

to spoof, and trivial to use?

- sticky labels (even if high tech) provide effective tamper detection?

- a mechanical tamper switch is serious security?

- cargo/shipment supply chains are secure?

- engineers understand security?

Chain of Custody for Devices (con’t)

14

Are your physical access control systems designed by

the sales guy, amateurs, or your cyber security people?

Do your locked doors have hinges on the outside?

Can someone open the door without using the access

control system and without it knowing?

Does your physical access control system know when an

employee has left the control area?

Are you under the mistaken impression that biometric

access control devices can’t be easily defeated? That

biometric signatures can’t be easily counterfeited?

Physical Access Control for Cyber

Questions Vulnerability Assessors Ask And You Should, Too

15

Questions Vulnerability Assessors Ask And You Should, Too

Do you have Role-Based Access Control, so

that access is halted INSTANTLY when

someone is promoted, given a new assignment,

or terminated?

Do you periodically review access control

privileges for all employees?

General Access Control

16

Questions Vulnerability Assessors Ask And You Should, Too

Is HR’s role in security objectively evaluated at least annually?

Does HR harm security instead of helping it?

If HR is indeed evil (likely), do managers, supervisors, & security

managers try to compensate?

Do you rely on the 80% rule (“listen, empathize, validate”) to mitigate

insider threats?

Do narcissists get their ego stroked on a regular basis?

Are there constraints on bully/harassing bosses?

Are retiring and terminated employees treated well? Is there a perp-walk

for terminated employees? Is there considerable HR glee at

firing employees?

HR & Insider Threat Mitigation

17

Questions Vulnerability Assessors Ask And You Should, Too

Are background checks on key personnel done periodically

and thoroughly, including interviewing acquaintances?

Do you do bribery anti-stings?

HR & Insider Threat Mitigation (con’t)

18

Questions Vulnerability Assessors Ask And You Should, Too

Do you exploit psychology research?

Sign a pledge of honesty at the top of documents, not the bottom.

Angry eye posters in critical areas.

Warn well-paid employees of the risk to themselves if they do

something unethical, but warn low-paid employees of the potential

harm to others.

Social influence for better security

Sunk-cost bias

Countermeasures to groupthink & to cognitive dissonance

Research on creativity

If someone has a security concern, including

about a fellow employee, can they

submit it anonymously? Does everybody know how?

Is it safe? Does anybody do it?

What happens when they do?

HR & Insider Threat Mitigation (con’t)

19

Questions Vulnerability Assessors Ask And You Should, Too

Is Security getting confused with

Control

Hassling/Threating Employees

Privacy or Safety

Inventory Management

Compliance & Auditing

Is high-tech confused with high-security?

Is your security awareness & social engineering

training effective? One-size-fits-all?

Security Culture & Management

20

Questions Vulnerability Assessors Ask And You Should, Too

Do you warn employees about what happened elsewhere

after a serious security incident?

Do people affected by security rules have input about them?

Do security rules get reviewed often?

Is there unwarranted faith in “layered security”?

Security Culture & Management (con’t)

21

Questions Vulnerability Assessors Ask And You Should, Too

Are employees told what security attacks look like, or just

given an unmotivated list of “things not to do”?

Are security rules and procedures motivated and justified?

Is security “accountability” mostly through disciplining,

firing, or scapegoating people?

Are awards and recognition given for good security

practices, or is security only about bad news?

Security Culture & Management (con’t)

22

Questions Vulnerability Assessors Ask And You Should, Too

Have a cyber monoculture?

Overlook the security benefits of OpenBSD, Linux, Mac OS

X, and iOS, especially for routine use?

Is your SOC your NOC?

How do regular employees recognize legitimate IT personnel

and instructions?

Use of 2-Factor Authentication?

Cyber Specific Issues

23

Watch Out for Compliance-Based Security:

Compliance Can Harm Security!

Rule of Thumb: About 30% of security rules, standards, and

guidelines in large organizations make security worse!

How

Creates a false sense of security

Wastes security resources, energy, and attention

on bureaucratic busywork/documentation/auditing

Supplants thinking and paying attention in favor of

formalistic mindlessness

Increases insider threat with all the extra auditors,

documenters, and checkers checking the

checkers

24

Watch Out for Compliance-Based Security:

Compliance Can Harm Security!How

Makes auditors the enemy, not adversaries

Engenders cynicism about security when rules are:- outdated

- unmotivated

- one-size-fits-all

- Security Theater

- only followed by the good guys

- ignorant of local conditions, culture, & vulnerabilities

- not given a sanity check by those affected

Makes security the enemy of employees &

productivity

25

Watch Out for Compliance-Based Security:

Compliance Can Harm Security!How

Used as an excuse not to do better when

minimum requirements are met

Following one standard or guideline is used as

an argument that security is good in other,

unrelated areas

Some standards are just bad (drafted by

vendors, special interests, and bureaucrats)

26

Security Metrics

Are things that are important getting measured,

or only things that are easy to measure?

Is only quantity measured but not quality?

Are your metrics mostly about costs, security

management, routine cyber activity,

and/or past incidents, not about security

effectiveness?

27

Some Unconventional Security Metrics

“What If?” exercises (+)

Transparency* (+)

Controversy & thoughtful pushback (+)

Count quality & quantity of:

28

Some Unconventional Security Metrics

Informal contact rate between non-security employees and

security employees. (+)

How frequently the grievance complaint resolution

process is used. (+)

Percent of security personnel for whom security is a career

choice. (+)

How often do terms like ”hackers”, “adversaries”,

“tamperers”, “counterfeiters”, and “bad guys” appear

in oral and written communication? (+)

29

Some Unconventional Security Metrics

Do minor security incidents or errors serve as statistical

precursors to serious incidents?

Employee turnover rates and Security personnel turnover

rates. (-)

Number of security changes

recently introduced. (+)

30

Marginal Analysis1. Continuously try incremental changes (real or theoretic) in your security

to see if it improves & risk decreases.

2. If it does, try more change in that ”direction”. If not, try another direction.

3. Occasionally try large changes to try to escape local minima in the risk

surface.

4. Somewhat counter-intuitively, change multiple parameters at once.

5. You have “pretty good security” if changes do not significantly lower the

risk.

.

31

Marginal Analysis Advantages

It’s easier to judge incremental changes in

security than total absolute security

effectiveness.

The emphasis on change may help encourage

proactive, flexible security, and overcome

security inertia, groupthink, cognitive dissonance,

complacency, and boredom..

In Summary

• Vulnerability Assessments are different & better than pen

testing, “Red Teaming”, security audits, threat assess-

ments, etc. Be sure you are doing VAs frequently (and not

something else that is getting confused with a VA)!

• VAers ask a lot of questions. You should, too.

• There are many possible unconventional security metrics

you could consider—including Marginal Analysis.

• Don’t rely on Compliance-Based Security! Compliance

and good Security are not that well correlated.

32

This presentation (with references)

and additional papers/talks are

available at:

http://rbsekurity.com

(Use “Papers & Talks” Tab)

For More Information...

http://jps.rbsekurity.com

34

Resources“Security Maxims”, https://tinyurl.com/y94wekyn

“Devil’s Dictionary of Security Terms”, http://rbsekurity.com/Papers/devils.pdf

“Compliance Versus Security”, Journal of Physical Security 10(1), 77-81 (2017), http://jps.rbsekurity.com

“Some Unconventional Security Metrics”, Journal of Physical Security 10(1), 82-85 (2017), http://jps.rbsekurity.com

“What Vulnerability Assessors Know That You Should, Too” Asia Pacific Security Magazine 50, 40-42, Aug/Sept 2013.

“Avoiding Shock and Awe”, Journal of Physical Security 9(1), 26-48 (2016) , http://jps.rbsekurity.com

wikiHow, “How to Validate Someone’s Feelings”, https://www.wikihow.com/Validate-Someone%27s-Feelings

”Time Series Analysis”, http://r-statistics.co/Time-Series-Analysis-With-R.html

Wikipedia, “Detrended Fluctuation Analysis”, https://en.wikipedia.org/wiki/Detrended_fluctuation_analysis.

R. Herold, “Do Compliance Requirements Help or Hurt Information Security?”, http://www.realtimepublishers.com/chapters/1699/esitcv1-13.pdf

J. Ross, Discover, “Signing a Form at the Top”, http://blogs.discovermagazine.com/80beats/2012/09/06/liar-liar-bottom-signer-signing-a-form-at-the-top-

leads-to-more-honest-answers/

J. Metcalfe, “Posters of Angry Eyes”, https://www.citylab.com/transportation/2013/04/posters-angry-eyes-actually-scare-bike-thieves/5420/

N Charky, “Eyeballs Have an Interesting Effect on Your Behavior”, https://archive.attn.com/stories/2854/eyeballs-effect-on-crime

M Hutson, “Rich People and Poor People Cheat for Different Reasons”,

https://www.thecut.com/2015/02/rich-and-poor-people-cheat-for-different-reasons.html

SS Wiltermuch, “Cheating More When the Spoils are Split”, Organizational Behavior and Human Decision Processes, 115(2), 157-168 (2001).

A Michel, “Psyber Securit”y, https://www.psychologicalscience.org/observer/psyber-security-thwarting-hackers-with-behavioral-science

R Anderson, “Psychology and Security Resource Page, https://www.cl.cam.ac.uk/~rja14/psysec.html

RG Johnston, “Security Sound Bites: Important Ideas About Security from Smart Ass, Dumb Ass, and Kick Ass Quotations”.

The Journal of Physical Security, http://jps.rbsekurity.com