Incident Response Workshop, PHDays 2014 1

Incident Response - PHDays IV

Alexander Sverdlov

Incident Response Workshop, PHDays 2014 3

Intro

• whoami – itso @ Bulgarian bank, @xorred, linkedin/in/asverdlov• Focus of the workshop: practice > theory

• Prevention²• Detection• Reaction

• Sessions plan: 4 hours, breaks ~10min every 50 min (if we run faster, will have time for discussions / coffee)

• Last slide contains the link to this presentation & other files

Incident Response Workshop, PHDays 2014 4

The boring stuff first

• http://csrc.nist.gov/publications/nistpubs/800-61rev2/SP800-61rev2.pdf - Computer Security Incident Handling Guide

• http://csrc.nist.gov/publications/nistpubs/800-83/SP800-83.pdf - Guide to Malware Incident Prevention and Handling

• Okay, you’ve read these 2… now what?

Incident Response Workshop, PHDays 2014 5

The boring stuff first

Incident Response Workshop, PHDays 2014 6

Prepare – step 1: situational awarenessWhat news sources do you inform yourself from on exploits, vulnerabilities, threats and 'what is going on now'?

• http://www.rootsecure.net/?p=secnews_rss_feeds

• https://twitter.com/search?q=%23dfir&src=typd #DFIR

• https://twitter.com/search?q=%23infosec&src=typd&f=realtime #INFOSEC

• https://www.us-cert.gov/ncas/current-activity

• http://www.reddit.com/r/netsec

• http://cert.europa.eu/cert/filteredition/en/CERT-LatestNews.html

• If a system you’re running is related to the above links (is vulnerable) – you might have an incident.

• Old school folks can still use RSS. It’s OK to live in the past!

Incident Response Workshop, PHDays 2014 7

Prepare - step 2: PPP (Policy, Plan, Procedures)• http://csrc.nist.gov/publications/nistpubs/800-61rev2/SP800-61rev2.pdf pages 7-9 +

http://technet.microsoft.com/en-us/library/cc184918.aspx

• Please do not use ready-made templates – they are and will be ineffective. Create your own from scratch and improve them with every incident / idea over time.

• Your policies / procedures will only get effective once you simulate or practice DFIR

• Do not forget to focus on authority in a visible place

“No Plan Survives The First Battle”

Incident Response Workshop, PHDays 2014 8

Prepare: step 3: Incident Prevention• Server hardening goes without question (look into DISA STIGs/SRGs)• But servers are on the inside with little to no external exposure… the most

exposed assets are people (and even more so, Admins), so:• Least Privilege Principle, 2 user accounts per Admin• Internal FW rules to block workstation traffic to most servers• SRP• EMET (4.1 or 5.0 if you’re feeling adventurous)• Automated updates for Flash, Java, Shockwave, PDF readers, Office, etc. Whitelisting their

addresses on your proxy (http://technet.microsoft.com/en-US/systemcenter/bb892875.aspx) • Whitelisted internet access for most users. Blocking certain sites / categories even for admins

(all of them). • Education (security awareness training) *including for their home computers*

Incident Response Workshop, PHDays 2014 9

Least Privilege Principle

• 2 users per admin account (1 limited, 1 local admin on their workstation), 1 limited user for everybody else

• *Explain why*

• Enforce the rule by random checks

• Consequences: Riots, hate, bypass attempts (admins are smarter than you and have the right to bypass Information Security Policies in their own minds). Involve HR if you have to, your organization does not need daredevils – it needs order. *note: if you didn’t explain why you’re enforcing this and what the risks are with proper examples, it’s your fault for having opposition. If you did explain well, non-compliance should have consequences.

• You’re not preventing malware. You’re preventing foreign code running in your domestic environment.

Incident Response Workshop, PHDays 2014 10

Internal FW rules

• Default firewall DENY even on the inside from any non-critical asset towards any critical asset

• Servers should be accessible only from the sysadmin group vlan(s)• Ideally servers should ONLY allow 2- or 3- factor authentication

Incident Response Workshop, PHDays 2014 11

Software Restriction Policy Example …that worksSoftware restriction policies: -Security Levels:--Default: Unrestricted-Additional Rules:--Path Rules:---%Temp%* (Disallowed)---%USERPROFILE%\AppData\Local\Temp* (Disallowed)---%USERPROFILE%\AppData\Local\Microsoft\Windows\INetCache\IE* (Disallowed)---%USERPROFILE%\AppData\Local\Microsoft\Windows\Temporary Internet Files\* (Disallowed)---%USERPROFILE%\Local Settings\Temp* (Disallowed)---E:\* (Disallowed)---F:\* (Disallowed)

Feel free to test and improve/improvise. The above stops executables from running from USB devices /Temp folders effectively blocking a lot of malware / exploits before they even have a chance to run. DidI mention this also solves the problem of someone receiving an archive with .exe(cutable) in it and opening it right from the attachment?

Incident Response Workshop, PHDays 2014 12

Software Restriction Policy Example …that works

For an additional guide (from the NSA) on SRP, check out:http://www.nsa.gov/ia/_files/os/win2k/application_whitelisting_using_srp.pdf andhttp://technet.microsoft.com/en-us/magazine/2008.06.srp.aspx to prevent the policies applying to local administrators: http://technet.microsoft.com/en-us/library/cc776536(v=ws.10).aspx - also, create a separate policy to disable the previous policy for users in Administrators / Domain administrators group. Next:Implementhttp://serverfault.com/questions/351535/how-can-i-stop-exes-being-ran-from-removeable-media-such-as-usb-drives

Consequences:Some software will break. Some installations will break. Deal with it, have exceptions, test before running this code / policy in production. AppLocker: if your environment allows it, go for it.

Incident Response Workshop, PHDays 2014 13

EMET (enhanced mitigation experience toolkit)• Don’t be a coward and set Maximum security settings!

• Disable the “Caller Mitigation” protection for allcrashing apps, distribute the correct XML using SCCMor a policy (read the user guide).

• There will be consequences for your helpdesk / administrators, there will be apps crashing and you will need to set exclusions. But it’s not that scary, you’ll only have to go through it once.

• 99.99% of all exploits will not be able to bypass EMET. Worth it.

Incident Response Workshop, PHDays 2014 14

EMET (enhanced mitigation experience toolkit)

Proper Config+ Demo

Incident Response Workshop, PHDays 2014 15

Internet Filtering (whitelisting)

• http://gaugusch.at/squid.shtml - in some cases it is possible to block IE from accessing external sites – if you can, do it.

• Implement malware / shellcode scanning at the proxy

• With a good webfilter, forbid most categories of websites, if you can afford the time, use whitelisting and only allow URLs after approval. After a while patterns will get filled and you will rarely get new URLs.

• Enforce No-Internet-For-Admins (NSA, pdf)

Incident Response Workshop, PHDays 2014 16

Safe Browsing

• Sirrix (Browser in a box)• Sandboxie (sadly dead, now Invincea)• Bufferzone Pro• Lastly – Comodo offers a sanbox capable of running a browser (

http://help.comodo.com/topic-72-1-451-5141-.html), for free

Incident Response Workshop, PHDays 2014 17

Proper Hardening

DISA SRGs/STIGs

NSA hardening guides

Incident Response Workshop, PHDays 2014 18

DDoS Prevention / Preparation

1. Block all ICMP/UDP traffic at the border firewall. On http://www.cymru.com/Documents/icmp-messages.html there is a list of types of ICMP messages you could allow, though, if you insist on having ICMP for your Internet-facing servers.

2. Block Bogons https://www.team-cymru.org/Services/Bogons/ - ingress. Also, for Ingress filtering, follow http://tools.ietf.org/html/rfc2267 to block incoming packets with forged/spoofed source addresses.

3. Disable directed IP broadcast - as per http://tools.ietf.org/html/rfc2644 - which will prevent your network from becoming an amplifying site. (points 1-3 can be seen in the book “Hacking Exposed, 7th edition”)

4. Make sure your company actively (this is for the marketing / PR people, make sure they get it) communicates with your customers in social networks. Your users must have been used already to receiving communication from you in various social sites - twitter, vk, facebook, linkedin, etc - so that when an attack comes, your company is not completely cut out from the world.

5. Do not buy DDoS protection services 'in advance'.

Incident Response Workshop, PHDays 2014 19

Human IDS/IPS

• IDS/IPS systems don’t work as well as human versions of them (build your own Matrix?)

• Educated personnel (yes, your sysadmins need the most training, however smart they are in making datacenters tick) is your best defense

• So how do you go about training them? Flyers / spam e-mails? Next-next-finish trainings? Powerpoint presentations?

Incident Response Workshop, PHDays 2014 20

Human IDS/IPS – thank you, DISA!• There is only one Security Awareness resource I can recommend: http

://iase.disa.mil/eta/cyberchallenge/launchPage.htm • If you ask nicely, they will give you the SCORM version• 30% of all malware detected usually comes from trained employees. Unless you don’t

educate them – then you miss 30% of all malware… • In addition: Smartphones / Tablets training: http

://iase.disa.mil/eta/smartphone_tablet_v2/launchpage.htm • Phishing training:

http://iase.disa.mil/eta/phishing_v2/phishing_v2/launchPage.htm• Social Networking training:

http://iase.disa.mil/eta/sns_v1/sn/launchPage.htm • + BONUS: http://cdsetrain.dtic.mil/cybersecurity/

Incident Response Workshop, PHDays 2014 21

Human IDS/IPS – thank you, DISA!

Incident Response Workshop, PHDays 2014 22

Require proof of completion

Incident Response Workshop, PHDays 2014 23

Human IDS/IPS – security awareness training

• Mandatory for: All administrative / IT / Management personnel

• Optional for all other employees

• Head of IT or CIO/CTO should hold every IT team member accountable to submit a test score on the security awareness training (either this or any other one)

Incident Response Workshop, PHDays 2014 24

Detecting Incidents

• Forget about your IDS/IPS (very few actually detect incidents and yours is most likely not among them). In fact, most companies get notified of incidents by external parties, not their security appliances. And still…

• Optimize your logging, centralize it, SIEM-ize it (in that order)

• Have NSM (network security monitoring) running in parallel to * else

• Have your coworkers forward you * suspicious e-mails, reward them

• Antiviruses detect less than 70% of all malware developed daily. And we’re talking about mass malware. They will NOT catch customized malware, wrapped and packed with love for your organization (unless their behavior analysis is really, really good – and popular AVs don’t have that).

• 3rd party service: ShadowServer!

Incident Response Workshop, PHDays 2014 25

Set your logs in order

• Do you know what you’re logging and why?• If anything fishy happens and is luckily logged, will you know?• What amount of your logs is pure junk and… why are you logging it?

• Get to know your Event log: export to csv, convert to xls, filter by EventID, filter our the useless events just to get a feel of it

Incident Response Workshop, PHDays 2014 26

Set your logs in order – Basic Audit Policy

This is dangerously useless.

Incident Response Workshop, PHDays 2014 27

Set your logs in order – Advanced Audit Policy

• Advanced Audit Policy Configuration FAQ http://technet.microsoft.com/en-us/library/ff182311(v=ws.10).aspx,

• HOWTO: http://technet.microsoft.com/en-us/library/dd408940(v=ws.10).aspx

Incident Response Workshop, PHDays 2014 28

Your logs are all ordered and shiny? Good.• http://www.nsa.gov/ia/_

files/app/Spotting_the_Adversary_with_Windows_Event_Log_Monitoring.pdf

• First 20+ pages (out of 48) are about centralized logging for a reason. Do it.

• http://www.octopussy.pm• http://graylog2.org/• Logstash + ES + Kibana • Go commercial if you’re commercial

Incident Response Workshop, PHDays 2014 29

NSM - SecurityOnion• My vote goes to SO over your favorite IDS/IPS appliance• Free as in Speech (you still pay with your time / storage)• Sexy UI (as much as a free tool can have, but still a lot)• As per their website: based on Ubuntu and contains Snort, Suricata, Bro, OSSEC,

Sguil, Squert, Snorby, ELSA, Xplico, NetworkMiner• HW requirements: at least 16 GB RAM, 2-5 TB of storage (for a few days of traffic

in a medium-sized org.), 4 CPUs minimum. • 2 books to ready by the author of the tool: http://

www.amazon.com/Richard-Bejtlich/e/B001IR3KOW/ref=dp_byline_cont_book_1

Incident Response Workshop, PHDays 2014 30

3rd party Incident Detection Services

• ShadowServer.org• https://www.shadowserver.org/wiki/pmwiki.php/Involve/GetReportsOnYourNetwork

• Report Types• The reporting service monitors and alerts the following activity:• Detected Botnet Command and Control servers• Infected systems (drones)• DDoS attacks (source and victim)• Scans• Clickfraud• Compromised hosts• Proxies• Spam relays• Open DNS Resolvers• Malicious software droppers and other related information.

Incident Response Workshop, PHDays 2014 31

3rd party Incident Detection Services• http://www.google.com/safebrowsing/alerts/ (need your own AS)

Safe Browsing Alerts for Network Administrators allows autonomous system (AS) administrators to register to receive Google Safe Browsing notifications. The goal is to provide network administrators with information of malicious content that is being hosted on their networks.

• Team Cymru TC Console - https://www.team-cymru.org/Services/TCConsole/ - no cost, *in most cases* (more info: https://www.team-cymru.org/Services/TCConsole/tcconsole_trifold.pdf) It is a good collaboration platform, if you collaborate it will be free for you.

• https://postmaster.live.com/snds/index.aspx - detect data coming from their network towards your network after verifying your AS. “By providing data such as mail traffic statistics seen by Windows Live Hotmail to IP block owners (ISPs, in a broad sense), organizations are empowered to prevent spam, viruses, and other malicious activity from originating from their IP space.”

• https://spyeyetracker.abuse.ch/index.php - mostly check your IP addresses / domains for c&c traffic towards c&c servers. Interesting statistic: Average SpyEye binary Antivirus detection: 27.94%

• https://www.team-cymru.org/Services/BINFeed/ - for banks and financial institutions, showing if malicious traffic contains any data related to that specific bank.

Incident Response Workshop, PHDays 2014 32

An incident has been detected.

Now what?

Most likely: 1. People doing sh*t they’re not supposed to (willingly and unwillingly, including bypassing your carefully placed

fences to defend them from the wolves (and it’s your fault))2. Equipment and software failing miserably3. People losing data4. Mass-distributed malware5. Phishing attacks (with or without exploits and malware)6. DDoS7. Least likely: APT (if they happen to you, well.. You better have had the previously mentioned defenses)

Incident Response Workshop, PHDays 2014 33

Being trigger-happy is BAD

• Gather your Management, Legal, PR and IT teams and together figure out the best way to respond to any incident, following and possibly deviating from the written policies / procedures when needed and agreed by everyone.

• Minor incidents involving *any* access to an employee workstation for analysis: inform your and their direct management. NEVER analyze suspicious data on your own.

Incident Response Workshop, PHDays 2014 34

Get your tool chest ready

• HW: 1x16GB USB 3.0, 1x64GB USB 3.0, 1x1(2)TB USB 3.0 external drive, optional: write-blocker (USB version + SATA), good camera, USB/PS2 hw keyloggers, live USB creator, … couple coffee mugs

• Having a DFIR workstation is a luxury not everyone has. At least keep an image(s) you can quickly deploy in your virtual environment (SIFT workstation works well, you could build your own as well, you will need a Windows image to go along with the Linux DFIR image)

• Network share having all your tools ready, as well as on the 3 USB devices mentioned above. Regularly sync them all.

• Policy, procedures, incident handling forms, your notebook, out-of-band-communication tools, place to organize meetings (physical or virtual),

• No helpdesk software to track incidents? http://bestpractical.com/rtir/ gets you covered. Free.

• CSIRT tool folder / Windows IR procedure contained in the archive with this presentation

• Comunication: Group Skype, Group Google Hangout, Google Docs with 2-factor auth pre-set-up. Anything secure and not connected to your possibly compromised network.

Incident Response Workshop, PHDays 2014 35

Gather evidence

• Remote / local RAM capture if you have to act blazingly fast• Mandiant Collector if you have the time (1-4 hours)• Remote / local pcap dump• HDD imaging (with or without a write blocker, just do with what you

have)• Photo / video evidence of the environment• While not forgetting to remove the suspect locally or remotely from

the machine• Interesting project: https://code.google.com/p/grr/

Incident Response Workshop, PHDays 2014 36

Demo: Windows IR Procedure step-by-step

Demo: Win-UFO

Demo: DEFT (DIGITAL EVIDENCE & FORENSIC TOOLKIT)

Demo: DART (indescribable boatload of awesomeness)

Incident Response Workshop, PHDays 2014 37

Containing / Eradicating incident consequencesBusiness <> Evidence <> Strategy <> Risk from incident <> Risk from not containing fast enough <> Risk from containing too fast

38

Simulation Scenarios (NIST 800-61rev2)1. Measures to prevent?

2. Which precursors to detect? Any of them could be used to detect before impact?

3. Indicators of an impact?

4. Additional tools you might need to detect it?

5. Who is involved in analysis and validation and how does it happen?

6. Who do you report it to?

7. Priorities during handling the incident?

8. Strategies to contain the incident and which are better than others?

9. What will happen if you cannot contain the incident?

10. Additional tools needed to respond?

11. Who is involved in responding to this incident?

12. Sources of evidence? How do you acquire it? Where do you store it and for how long?

13. Who is at the ‘lessons learned’?

14. How do you prevent it in the future?

15. How do you improve detection in the future?

Incident Response Workshop, PHDays 2014

Incident Response Workshop, PHDays 2014 39

Calculating the cost of the incident

• cybertab.boozallen.com – once again, free. Completely anonymous (optional), accurate guideline & report on the cost of an incident.

Incident Response Workshop, PHDays 2014 40

Questions?

Incident Response Workshop, PHDays 2014 41

Download Link for the accompanying materials:

http://goo.gl/zCk6tO

Presentation: http://goo.gl/Dm8oBJ