from&tool&to&team& member:&...

Copyright © 2014 Splunk Inc.

George Starcher Security Engineer, Peak Hos6ng

From Tool to Team Member: Controlling Systems with Splunk Alert Scripts

About Me !   George Starcher, Informa6on Security Engineer ‒  CISSP, Splunk Cer6fied Knowledge Manager and Splunk

Cer6fied Administrator !   Splunk IRC Channel !   Looking to kick off a Nashville, TN – Splunk User Group •  www.georgestarcher.com •  www.peakhos6ng.com

2

Agenda

!   Splunk from Tool to a Team Member !   How it Works !   GeQng into the Code ‒  Alert Script to Intrusion Preven6on System Control ‒  Alert Script to X-‐ARF Abuse Repor6ng

3

“Using Alert Scripts to take ac6on on our behalf, we can transform Splunk from a tool to a team member.”

Splunk from Tool to Team Member !   Manual Abuse Scanning Process ‒  Reviewed SSH, RDP, VNC etc daily ‒  Consumed 30-‐45 minutes per day ‒  Permanent blacklist entries

!   Moved to automated process ‒  Scheduled Splunk Searches driven by any log source ‒  Greatly reduced 6me and sta6c blacklist maintenance ‒  Web Services (REST) calls to the IPS

4

Splunk from Tool to Team Member

5

Outlook Web Access – Phishers

Started Feb 10, 2014

•  Blocked for any access from Nigeria every 5 minutes

Expanded Mul6 Country Feb 15, 2014

•  Blocked for combina6on from certain countries & a lookup table of hosted providers

Feb 17, 2014

•  No6ced unexpected Exchange OWA from Nigeria

Splunk from Tool to Team Member

6

Outlook Web Access – Phishers

Single User by src_ip_country:

Hosted Lookup users by src_ip:

How it Works

How it Works

8

Intrusion Preven6on Appliance

How it Works

9

How it Works

10

h8p://blogs.splunk.com/2011/03/15/storing-‐encrypted-‐credenCals/

h8p://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automaCng-‐control/

How it Works

11

!   Setup a service account to own the Alert Searches: svc-‐alert !   Create a role just for the alert account !   That role must have ‘admin_all_objects’ !   The role must have access to all indexes that might have the data for the scheduled search alert

Alert Service Account

How it Works

12

Alert Script in Ac6on

13

!   Avoids manual repor6ng !   Ensures 6mely ac6on !   Consistent Repor6ng Format !   Accurate Evidence Data !  Works around the clock and doesn't need coffee

X-‐ARF Abuse Repor6ng


14



15


Crawling into the Code

@SplunkDev Team -‐ THANKS!!

17

@gblock -‐ Glenn Block

@damiendallimore -‐Damien Dallimore

David Noble -‐ Twiner App

Where Can You Get The Code?

18

! Github Repository ‒  hnps://github.com/georgestarcher/Splunk-‐Alert ‒  General Intrusion Preven6on System Example Code ‒  Google Spreadsheet Upload Code ‒  X-‐ARF Abuse Repor6ng Code

!   The Google Spreadsheet Example ‒  hnp://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automa6ng-‐control/

Arguments Sent to Alert Scripts

19

h8p://docs.splunk.com/DocumentaCon/Splunk/6.1.3/Alert/Configuringscriptedalerts

!   SPLUNK_ARG_0 Script name !   SPLUNK_ARG_1 Number of events returned !   SPLUNK_ARG_2 Search terms !   SPLUNK_ARG_3 Fully qualified query string !   SPLUNK_ARG_4 Name of report !   SPLUNK_ARG_5 Trigger reason (for example, "The number of events was greater than 1")

!   SPLUNK_ARG_6 Browser URL to view the report !   SPLUNK_ARG_8 File in which the results for this search are stored (contains raw results)

The Code Modules – IPS

20

!   creden6alsFromSplunk.py ‒  A Python class to fetch the saved service account

!   targetlist.py ‒  The Python class for data to be handled

!   ips.py ‒  The Python class for an IPS rest API interface

!   alert_script.py ‒  The main Python alert script

creden6alsFromSplunk.py

21

!   A re-‐usable Python class to fetch stored user creden6als from Splunk !   Provide the app where creden6aled is stored: splunkapp !   Provide the purpose name used when saving the creden6als: realm !   Provide the username to be retrieved: username !   Call the getPassword method

creden6alsFromSplunk.py

22

# Define the source in Splunk for the stored credenCal splunkapp = "myadmin" realm = 'ips' username = 'splunk'

# Get the stored credenCal from Splunk try: ipsCredenCal.getPassword(sessionKey) except ExcepCon, e: logError("Splunk CredenCal Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_SPLUNK_AUTH)

# Define the ips connecCon ipsCredenCal = credenCal(splunkapp,realm,username)

from alert_script.py

targetlist.py

23

!   A simple Python class for a single column list of source IPs !   Populated by the alert search returning only source IPs !   Takes argument of path to the search results to load the list

# Obtain the path to the alert events compressed file and load the search results to the list alertEventsFile = os.environ['SPLUNK_ARG_8']

try: alertTargetList = targetlist(alertEventsFile) except ExcepCon, e: logError("Target File Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_TARGET_FILE)


ips.py

24

!   An example Python class to interface with our Intrusion Detec6on System Rest API

!   Setup and retrieve the creden6al from splunk: ipsCredenCal !   Provide the IPS quaran6ne policy name: policy_name !   Provide IP address of the IPS management Interface: ips_ip !   Ac6vate the IPS rest connec6on object !   Loop through the alertTargetList having the IPS quaran6ne each IP

Make your Own REST API wrapper class to control other systems

ips.py

25

# AcCve the ips connecCon object try: ssh_ips = ips(ips_ip,ipsCredenCal.username,

ipsCredenCal.password,policy_name) except ExcepCon, e: logError("IPS Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)


alert_script.py

26

!   The main script called by Splunk for our alert search !   Imports all our classes !   Parses the sessionKey !   Connects to our IPS !   Pulls in the search result list of IP addresses !   Loops through the IP list and tells the IPS to quaran6ne them

alert_script.py

27

The Hash Bang: #!/opt/splunk/bin/python

# QuaranCne each source ip in the alert results table

for address in alertTargetList.targetlist: try: ssh_ips.addQuaranCne(address) except ExcepCon, e: logError("IPS QuaranCne Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)

# Obtain the Splunk authenCcaCon session key … # Adjust the returned sessionKey text based on Splunk version …

Extended Abuse Repor6ng -‐ X-‐ARF

28

!   Much more complex code !   Search results driving results is a table of data not a simple IP list !   Pulls email seQngs from Splunk !   Builds the email body using the Python Mako template (mail merge to search results)

!   Improved alert script ac6on logging sending into index=_internal !   Anaches Alert Event Search results from Splunk REST API Calls

h8p://www.x-‐arf.org/

BONUS

The Code Modules -‐ X-‐ARF

29

!   abuselist.py ‒  The data to be handled

!   emailSplunkXARF.py ‒  A python class to fetch the saved service account

! xarf-‐abuse.tmpl ‒  Abuse report Email mako template

!   alert_to_xarf.py ‒  The main alert script

abuselist.py

30

!   Method getEvidence holds the evidence search executed against the Splunk REST API

!   This method also manipulates the earliest/latest 6mestamp coming from the search results automa6cally to go into the detail evidence search

emailSplunkXARF.py !   Method getMailSeQngs is Splunk REST API call to fetch the seQngs from your Splunk server

alert_to_xarf.py

31

!   All the X-‐ARF values are at the top of the script !   Method getSplunkVersion gets the running Splunk version from the REST API to help auto adjust the sessionKey

!   Method getSplunkUser gets the username the Alert executed under from Splunk needed for the evidence search fetch

!   Logging writes with proper 6mestamp GMT to $SPLUNK_HOME/var/log/splunk/…

You could use this to make your own highly customized alert email based on search results

Thank You!

32

Other resources Splunk IRC ( EFNet #splunk ) Splunk Answers ( hnp://answers.splunk.com ) Splunk community wiki ( hnp://wiki.splunk.com ) hnp://www.georgestarcher.com/ hnp://blog.splunk.com/ hnp://www.meetup.com/Splunk/Nashville-‐TN/

Other “must-‐see” .conf 2014 presentaCons !   Avoid the SSLippery Slope of Default SSL -‐ Duane Waddle and George Starcher !   In Depth With Deployment Server -‐ Dave Shpritz, Aplura !   Using Lesser Known Commands in Splunk Search Processing Language (SPL) -‐ Kyle Smith, The Hershey Company

!   Masters of IRC -‐ panel talk on the Splunk Community Stage

THANK YOU

from&tool&to&team& member:&...

Documents