© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive and best practices Sunny Suen Managing Principal, Asia ESP Solutions Consulting
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
FlexConnector deep dive and best practices
Agenda FlexConnector deep dive • Customized event feeding options • Advanced topics in FlexConnector development
FlexConnector best practices • Essential steps on FlexConnector configuration &
development • Best Practice of FlexConnector submission to improve
– Maintainability – Readability – Efficiency – Accuracy
Target audience Go through FlexConnector training and documentation Have created FlexConnector in practical environment
Reference Documentation • FlexConnector Development Guide • ArcSight Categorization Technical Note • Other SmartConnector Configuration Guide
Training • FlexConnector Training
Introduction
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive
Customized event feeding options
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
SmartConnector architecture
ArcSight message Event ID Event name Base event count Detect time Target address Source address Device address
Event data
IDS
Firewall
Unix Syslog
Event data
Event data
Event data
SmartAgent
SmartAgent
SmartAgent
Event data (ArcSight message)
Event data (ArcSight message)
Event data (ArcSight message) ArcSight manager
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
Consideration for event feeding
SmartConnector • An application that collects raw events from security devices,
processes them into HP ArcSight security events, and transports them to destination devices.
• SmartConnectors are generally one of the following types – File Connectors – Database Connectors – API Connectors – SNMP Connectors – Microsoft Windows Event Log Connectors – SyslogConnectors – ScannerConnectors – FlexConnectors – ModelConnectors
FlexConnector • The FlexConnector framework is a software development
kit (SDK) that lets you create a SmartConnector tailored to the devices on your network and their specific event data.
• The available FlexConnectors are: – Logfile FlexConnector (fixed-format) – Regex FlexConnector (variable-format) – Database FlexConnectors – SNMP FlexConnector – Syslog FlexConnector – XML FlexConnector – Scanner FlexConnector – REST FlexConnector – Key-value FlexConnector (via Logfile/Regex
FlexConnector)
SmartConnector vs. FlexConnector
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
Do we need FlexConnector?
Consideration for event feeding
First, capture the requirement • Capture all device log details
– Vendor Name (e.g. ABCDE Technologies) – Product Name (e.g. ABCDE Web Server) – Software/Firmware version (e.g. version X.X) – Log Type (e.g. flat file) – Log format / transport (Free format text log) – Log rotation scheme (Daily; filename containing event log date: AccessLog_yyyyMMdd.log) – Event Type (e.g. access audit log)
• Match supported device list of SmartConnector – If not in the list, seek for following options…
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
Do we need FlexConnector?
Consideration for event feeding
Options for customized event feeding • FlexConnector Development
– For a complete development of parser/categorization on all required events – Identify FlexConnector format / transport
• Reuse of SmartConnector parser/categorization – Partial development for unparsed/uncategorized events – Identify the similar type of parser/categorization
• Map files/External Mapper – For further interpretation some event values – E.g. elaborating department name “HR” to “Human Resource Department”
• Common Event Format (CEF) – Add/modify application log format as CEF output
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8
Type of FlexConnectors
FlexConnector development
• Logfile FlexConnector (fixed-format) • Regex FlexConnector (variable-format) • Database FlexConnectors • SNMP FlexConnector • Syslog FlexConnector • XML FlexConnector • Scanner FlexConnector • REST FlexConnector • Key-value FlexConnector (via Logfile/Regex FlexConnector)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
Reuse of SmartConnector parser
When I need this? To reuse (convert) standard SmartConnector’s parser of the required log transport type, such as • Convert a file-based FlexConnector to folder-based FlexConnector • A file-based FlexConnector is wrapped by syslog transport
How to use it? • Identify the folder/file name of standard parser file (from aup)
unzip -l {$Connector}/current/system/agent/arcsightagents.aup
– e.g. apache/apache_access_file • Scenario 1: Converting standard file reader Connector to multi-folder Connector
– Configure Multi-Folder FlexConnector to assign the configfile as the standard parser path agents[0].foldertable[0].configfile=apache/apache_access_file
• Scenario 2: Converting standard file reader Connector to syslog transport – Create Syslog FlexConnector with following extraprocessor statement extraprocessor[0].type=regex
extraprocessor[0].filename=apache/apache_access_file
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
Reuse of SmartConnector categorization
When I need this? To reuse the standard categorization file of one supported device to the FlexConnector How to use it? • Check in the agent.log on the categorization file being used • Identify the folder/file name of standard categorization file (from aup)
unzip -l {$Connector}/current/system/agent/arcsightagents_{date-version}.aup
– e.g. apache/apache.csv • Create additional categorization file, e.g. newvendor/newproduct.csv • Include the folder/file name of standard categorization file into the link file, e.g.
– Create a file newvendor/newproduct.link.csv in the acp/categorizer/current folder – Add following lines into newproduct.link.csv • /apache/apache.csv • /newvendor/newproduct.csv
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11
Map file revisit
When I need this? To map additional values from event field(s) to assign new values
How to use it? • create map.n.properties (where is 1,2,3,etc in sequence) • put getter and setter mapping on the first line of the map.n.properties file, such as
– Line 1: Getters and setters • event.deviceHostName,set.event.deviceCustomString1
– Line 2 onwards: value mapping • Host1,HR Dept • Host2,FIN Dept
Type of getters • Exact match: Header Row=event.deviceHostName, data=Host1 • Range: Header Row=range.event.destinationPort, data=10000-19999 • Regex: Header Row=regex.event.deviceHostName, data=HR.*
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
External mapper
When I need this? Perform external database query on event field for additional mapping information How to use it? • Check the agent URI (with 2 ‘equal sign (==)’ suffices) • Create the folder: user/agent/extmap/{agent URI} • Create the external mapper file extmap.n.properties (where n=1,2,3… in sequence) into this folder
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
(Continued)
External mapper
• Add following lines into the extmap.n.properties type=sql field.getter=deviceAddress field.setter.count=1 field.setter[0]=deviceCustomString1 #field.addrs.as.numbers=true jdbc.class=org.gjt.mm.mysql.Driver jdbc.url=jdbc:mysql://localhost:3306/threatIntel jdbc.username=intel jdbc.password=OBFUSCATE.4.8.1:LOX6GXaJ+5imr6M1wmwkNg== jdbc.query=select address, threatVector from watchList WHERE address in (?\u0000?)
• The password has can be created by this command ./arcsight agent runjava com.arcsight.agent.loadable._ExternalMapperComponent -p 'arcsight'
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14
Type of FlexConnectors
CEF format
When I need this? To adapt the application log to ArcSight CEF format. This can eliminate the FlexConnector parser maintenance afterwards How to use it? • Obtain the “Common Event Format” Documentation • Adapt the application log output to CEF format • Create event categorization file based for all possible event types
HP ArcSight Common Event Format
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive
Advanced FlexConnector topics
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
Advanced topics in FlexConnector development
Scope • Advanced techniques commonly encountered in practical environment • Not currently discussed in documentation in details
Topics • Fragmented event lines • Character encoding
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17
Fragmented event lines
When I need this? Some devices send single event in multiple log lines To merge the information of all the related events into a single one Handling options • multiline.regex • Regex (?s) • Line.ignore.regex • Event merging
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18
Event merging
When I need this? In some instances the events sent by the device will not necessarily be close together, there could be events that will be sent in between other events, such as
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
Event merging
How to use it? Use merge parameters:
merge.count=1 merge[0].pattern.count=2 merge[0].pattern[0].token=NAME1 merge[0].pattern[0].regex=(BIND|UNBIND|MOD|RESULT) merge[0].pattern[1].token=NAME2 merge[0].pattern[1].regex=(BIND|UNBIND|MOD|RESULT)2 merge[0].starts.count=1 merge[0].starts[0].token=NAME3 merge[0].starts[0].regex=(BIND|UNBIND|MOD)
merge[0].ends.count=3 merge[0].ends[0].token=NAME4 merge[0].ends[0].regex=RESULT merge[0].ends[1].token=NAME5 merge[0].ends[1].regex=RESULT2 merge[0].ends[2].token=NAME6 merge[0].ends[2].regex=RESULT3 merge[0].timeout=60000 merge[0].id.tokens=conn|msgId merge[0].id.tokens.delimiter=| merge[0].sendpartialevents=true merge[0].capacity=100
(Continued)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
Character encoding
When I need this? The raw log data contains non-ASCII characters
How to use it? Configuration: agent.properties • The Multi Folder Follower FlexConnector
– agents[0].foldertable[0].encoding=UTF-16LE • SNMP FlexConnector
– snmp.charset={Your character set for the foreign language} (such as snmp.charset=big5)
Configuration: JVM option • For other FlexConnector (such as Syslog FlexConnector)
– In ${CONNECTOR}/current/bin/scripts/connectors.sh (or connectors.bat), append following options in the “ARCSIGHT_JVM_OPTIONS”:
-Dfile.encoding=(character encoding) (such as -Dfile.encoding=gb2312)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
(Continued)
Character encoding
Development: regex – main / submessage regex statement • convert non-ascii/multi-byte characters into unicode notation, such as \u7528\u6236 (Chinese
characters of ‘User’) • Conversion tool: native2ascii (included in JDK distribution)
• put unicode notation in regex statement
$ native2ascii 用戶 \u7528\u6236
regex=\\u7528\\u6236: (\\S+)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector best practices
Files to be created/configured in FlexConnector development
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 23
Folder structure
FlexConnector Configuration ${ArcSight Connector Home}/current/user/agent User Development Folders • flexagent: parser • acp: event categorization • fcp: standard parser / parser override • map: map file • extmap: external mapper • lib: jdbc driver
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24
agent.properties
Configuration
Location under {$Connector_Home}/current • user/agent
When I need it? To tune the advanced configuration parameters, such as: • File Rotation • Filename extractor • Subagent list
Refer to agent.default.properties for more options How to use it? • For Software Connector, modify the file directly in user/agent/agent.properties • For Connector Appliance, use “Diagnostic Wizard” and select agent.properties
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25
parser (*.properties)
Development
Location under {$Connector_Home}/current • user/agent/flexagent
Sections in properties file • Parser configuration • Token Declaration • ArcSight event field assignment • Severity mapping • Conditional mapping • Extraprocessor
Point to Note • Proper comment should be required
– Sample messages – Message group – Section heading
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 26
File extension and Location
Development
flexagent/ • Log-file: <product_name>.sdkfilereader.properties • Regex Log-file: <product_name>.sdkrfilereader.properties • Regex Folder Log-file: <product_name>.sdkrfilereader.properties • XML Folder Log-file: <product_name>.xqueryparser.properties • Time-based Database: <product_name>/<table_name>.sdktbdatabase.properties • ID-based Database: <product_name>/<table_name>.sdkibdatabase.properties • Multi-Database: <product_name>/table <n>/<table_name>.sdktbdatabase.properties • Syslog: syslog/<product_name>.subagent.sdkrfilereader.properties • SNMP: <vendor>/sdksnmp.#.snmptrap.properties (where # = trap type) • Scanner (Normal Text): <vendor>.scanner.sdkrfilereader.properties • Scanner (XML): <vendor>.scanner. xqueryparser.properties • Scanner (Database): <vendor>. sdkdatabase.properties • REST: <product_name>.jsonparser.properties
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27
Categorization (*.csv)
Development
Location under {$Connector_Home}/current • user/agent/acp/categorizer/current/<device_vendor>/<device_product>.csv
File format Getters and setters • Exact match, range, regex getters are all supported Single categorization file: vendor1/product1.csv Multiple categorization file: vendor1/product1.link.csv, vendor1/product.csv and vendor1/product.0.csv
Point to Note File name capitalization • All folder and file names are in small letters • Space or non-alphanumeric letters are converted into underscore ‘-’, such as
– AS/400 -> as_400 – Microsoft Windows -> microsoft_windows
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 28
Categorization (*.csv) (Continued)
Development
Point to Note (continued) • Try using Regex getter and put some catch all to enhance the categorization result, such as
– Logon.* (For all undefined event type with “Logon” prefix) – .* (For all other undefined event type)
• Required categorization fields regex.event.deviceEventClassId,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categoryDeviceType,set.event.categoryOutcome,set.event.categorySignificance,set.event.agentSeverity
• Obtain the ArcSight Categorization Technical Note
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29
Key field assignment
Topics Time fields: • deviceReceiptTime, startTime, endTime
Device name for categorization link: • deviceVendor / deviceProduct
Event type: • name/deviceSeverity/deviceAction/deviceEventClassId
Custom fields • Use deviceCustom* and avoid flex* • flex* fields are reserved for console mapping to additional data
Source-destination vs attacker-target • Only use source/destination pair (e.g. sourceUserName/destinationUserName) • attacker/target pair: result of correlation, not to be assigned in FlexConnector
Assignable fields (list of assignable fields in FlexConnector Developer’s Guide)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30
Local testing (CSV/CEF output)
When I need this? To test and debug the parser/categorization in the development phase, it will be more efficient to verify locally rather than submitting to logger and ESM How to use it? • For connector destination, choose CEF log file or CSV log file • Select header row to help readablility • For CSV log file, you can choose the required fields as output
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 31
Packaging
For submission of FlexConnector for production usage, we recommend: • Single Package file • Folder structure following Connector installation standard • Put device vendor/device product/FlexConnector version number in the package name
Example Package: $deviceProduct_flexconnector_vx.x_yyyy-mm-dd.zip Content folder structure: acp/
fcp/
flexagent/
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32
Documentation
Highly recommend to provide following documentation in the FlexConnector files for ongoing maintenance: • Parser comments • Configuration Documentation
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 33
Please fill out a survey. Hand it to the door monitor on your way out.
Thank you for providing your feedback, which helps us enhance content for future events.
Session TB3033 Speaker Sunny Suen
Please give me your feedback
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you