flexconnector deep dive and best practices name base event count detect time target address source...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

FlexConnector deep dive and best practices Sunny Suen Managing Principal, Asia ESP Solutions Consulting

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2

FlexConnector deep dive and best practices

Agenda FlexConnector deep dive • Customized event feeding options • Advanced topics in FlexConnector development

FlexConnector best practices • Essential steps on FlexConnector configuration &

development • Best Practice of FlexConnector submission to improve

– Maintainability – Readability – Efficiency – Accuracy

Target audience Go through FlexConnector training and documentation Have created FlexConnector in practical environment

Reference Documentation • FlexConnector Development Guide • ArcSight Categorization Technical Note • Other SmartConnector Configuration Guide

Training • FlexConnector Training

Introduction


FlexConnector deep dive

Customized event feeding options


SmartConnector architecture

ArcSight message Event ID Event name Base event count Detect time Target address Source address Device address

Event data

IDS

Firewall

Unix Syslog

Event data

Event data

Event data

SmartAgent

SmartAgent

SmartAgent

Event data (ArcSight message)

Event data (ArcSight message)

Event data (ArcSight message) ArcSight manager


Consideration for event feeding

SmartConnector • An application that collects raw events from security devices,

processes them into HP ArcSight security events, and transports them to destination devices.

• SmartConnectors are generally one of the following types – File Connectors – Database Connectors – API Connectors – SNMP Connectors – Microsoft Windows Event Log Connectors – SyslogConnectors – ScannerConnectors – FlexConnectors – ModelConnectors

FlexConnector • The FlexConnector framework is a software development

kit (SDK) that lets you create a SmartConnector tailored to the devices on your network and their specific event data.

• The available FlexConnectors are: – Logfile FlexConnector (fixed-format) – Regex FlexConnector (variable-format) – Database FlexConnectors – SNMP FlexConnector – Syslog FlexConnector – XML FlexConnector – Scanner FlexConnector – REST FlexConnector – Key-value FlexConnector (via Logfile/Regex

FlexConnector)

SmartConnector vs. FlexConnector


Do we need FlexConnector?


First, capture the requirement • Capture all device log details

– Vendor Name (e.g. ABCDE Technologies) – Product Name (e.g. ABCDE Web Server) – Software/Firmware version (e.g. version X.X) – Log Type (e.g. flat file) – Log format / transport (Free format text log) – Log rotation scheme (Daily; filename containing event log date: AccessLog_yyyyMMdd.log) – Event Type (e.g. access audit log)

• Match supported device list of SmartConnector – If not in the list, seek for following options…


Do we need FlexConnector?


Options for customized event feeding • FlexConnector Development

– For a complete development of parser/categorization on all required events – Identify FlexConnector format / transport

• Reuse of SmartConnector parser/categorization – Partial development for unparsed/uncategorized events – Identify the similar type of parser/categorization

• Map files/External Mapper – For further interpretation some event values – E.g. elaborating department name “HR” to “Human Resource Department”

• Common Event Format (CEF) – Add/modify application log format as CEF output


Type of FlexConnectors

FlexConnector development

• Logfile FlexConnector (fixed-format) • Regex FlexConnector (variable-format) • Database FlexConnectors • SNMP FlexConnector • Syslog FlexConnector • XML FlexConnector • Scanner FlexConnector • REST FlexConnector • Key-value FlexConnector (via Logfile/Regex FlexConnector)


Reuse of SmartConnector parser

When I need this? To reuse (convert) standard SmartConnector’s parser of the required log transport type, such as • Convert a file-based FlexConnector to folder-based FlexConnector • A file-based FlexConnector is wrapped by syslog transport

How to use it? • Identify the folder/file name of standard parser file (from aup)

unzip -l {$Connector}/current/system/agent/arcsightagents.aup

– e.g. apache/apache_access_file • Scenario 1: Converting standard file reader Connector to multi-folder Connector

– Configure Multi-Folder FlexConnector to assign the configfile as the standard parser path agents[0].foldertable[0].configfile=apache/apache_access_file

• Scenario 2: Converting standard file reader Connector to syslog transport – Create Syslog FlexConnector with following extraprocessor statement extraprocessor[0].type=regex

extraprocessor[0].filename=apache/apache_access_file


Reuse of SmartConnector categorization

When I need this? To reuse the standard categorization file of one supported device to the FlexConnector How to use it? • Check in the agent.log on the categorization file being used • Identify the folder/file name of standard categorization file (from aup)

unzip -l {$Connector}/current/system/agent/arcsightagents_{date-version}.aup

– e.g. apache/apache.csv • Create additional categorization file, e.g. newvendor/newproduct.csv • Include the folder/file name of standard categorization file into the link file, e.g.

– Create a file newvendor/newproduct.link.csv in the acp/categorizer/current folder – Add following lines into newproduct.link.csv • /apache/apache.csv • /newvendor/newproduct.csv


Map file revisit

When I need this? To map additional values from event field(s) to assign new values

How to use it? • create map.n.properties (where is 1,2,3,etc in sequence) • put getter and setter mapping on the first line of the map.n.properties file, such as

– Line 1: Getters and setters • event.deviceHostName,set.event.deviceCustomString1

– Line 2 onwards: value mapping • Host1,HR Dept • Host2,FIN Dept

Type of getters • Exact match: Header Row=event.deviceHostName, data=Host1 • Range: Header Row=range.event.destinationPort, data=10000-19999 • Regex: Header Row=regex.event.deviceHostName, data=HR.*


External mapper

When I need this? Perform external database query on event field for additional mapping information How to use it? • Check the agent URI (with 2 ‘equal sign (==)’ suffices) • Create the folder: user/agent/extmap/{agent URI} • Create the external mapper file extmap.n.properties (where n=1,2,3… in sequence) into this folder


(Continued)

External mapper

• Add following lines into the extmap.n.properties type=sql field.getter=deviceAddress field.setter.count=1 field.setter[0]=deviceCustomString1 #field.addrs.as.numbers=true jdbc.class=org.gjt.mm.mysql.Driver jdbc.url=jdbc:mysql://localhost:3306/threatIntel jdbc.username=intel jdbc.password=OBFUSCATE.4.8.1:LOX6GXaJ+5imr6M1wmwkNg== jdbc.query=select address, threatVector from watchList WHERE address in (?\u0000?)

• The password has can be created by this command ./arcsight agent runjava com.arcsight.agent.loadable._ExternalMapperComponent -p 'arcsight'


Type of FlexConnectors

CEF format

When I need this? To adapt the application log to ArcSight CEF format. This can eliminate the FlexConnector parser maintenance afterwards How to use it? • Obtain the “Common Event Format” Documentation • Adapt the application log output to CEF format • Create event categorization file based for all possible event types

HP ArcSight Common Event Format


FlexConnector deep dive

Advanced FlexConnector topics


Advanced topics in FlexConnector development

Scope • Advanced techniques commonly encountered in practical environment • Not currently discussed in documentation in details

Topics • Fragmented event lines • Character encoding


Fragmented event lines

When I need this? Some devices send single event in multiple log lines To merge the information of all the related events into a single one Handling options • multiline.regex • Regex (?s) • Line.ignore.regex • Event merging


Event merging

When I need this? In some instances the events sent by the device will not necessarily be close together, there could be events that will be sent in between other events, such as


Event merging

How to use it? Use merge parameters:

merge.count=1 merge[0].pattern.count=2 merge[0].pattern[0].token=NAME1 merge[0].pattern[0].regex=(BIND|UNBIND|MOD|RESULT) merge[0].pattern[1].token=NAME2 merge[0].pattern[1].regex=(BIND|UNBIND|MOD|RESULT)2 merge[0].starts.count=1 merge[0].starts[0].token=NAME3 merge[0].starts[0].regex=(BIND|UNBIND|MOD)

merge[0].ends.count=3 merge[0].ends[0].token=NAME4 merge[0].ends[0].regex=RESULT merge[0].ends[1].token=NAME5 merge[0].ends[1].regex=RESULT2 merge[0].ends[2].token=NAME6 merge[0].ends[2].regex=RESULT3 merge[0].timeout=60000 merge[0].id.tokens=conn|msgId merge[0].id.tokens.delimiter=| merge[0].sendpartialevents=true merge[0].capacity=100

(Continued)


Character encoding

When I need this? The raw log data contains non-ASCII characters

How to use it? Configuration: agent.properties • The Multi Folder Follower FlexConnector

– agents[0].foldertable[0].encoding=UTF-16LE • SNMP FlexConnector

– snmp.charset={Your character set for the foreign language} (such as snmp.charset=big5)

Configuration: JVM option • For other FlexConnector (such as Syslog FlexConnector)

– In ${CONNECTOR}/current/bin/scripts/connectors.sh (or connectors.bat), append following options in the “ARCSIGHT_JVM_OPTIONS”:

-Dfile.encoding=(character encoding) (such as -Dfile.encoding=gb2312)


(Continued)

Character encoding

Development: regex – main / submessage regex statement • convert non-ascii/multi-byte characters into unicode notation, such as \u7528\u6236 (Chinese

characters of ‘User’) • Conversion tool: native2ascii (included in JDK distribution)

• put unicode notation in regex statement

$ native2ascii 用戶 \u7528\u6236

regex=\\u7528\\u6236: (\\S+)


FlexConnector best practices

Files to be created/configured in FlexConnector development


Folder structure

FlexConnector Configuration ${ArcSight Connector Home}/current/user/agent User Development Folders • flexagent: parser • acp: event categorization • fcp: standard parser / parser override • map: map file • extmap: external mapper • lib: jdbc driver


agent.properties

Configuration

Location under {$Connector_Home}/current • user/agent

When I need it? To tune the advanced configuration parameters, such as: • File Rotation • Filename extractor • Subagent list

Refer to agent.default.properties for more options How to use it? • For Software Connector, modify the file directly in user/agent/agent.properties • For Connector Appliance, use “Diagnostic Wizard” and select agent.properties


parser (*.properties)

Development

Location under {$Connector_Home}/current • user/agent/flexagent

Sections in properties file • Parser configuration • Token Declaration • ArcSight event field assignment • Severity mapping • Conditional mapping • Extraprocessor

Point to Note • Proper comment should be required

– Sample messages – Message group – Section heading


File extension and Location

Development

flexagent/ • Log-file: <product_name>.sdkfilereader.properties • Regex Log-file: <product_name>.sdkrfilereader.properties • Regex Folder Log-file: <product_name>.sdkrfilereader.properties • XML Folder Log-file: <product_name>.xqueryparser.properties • Time-based Database: <product_name>/<table_name>.sdktbdatabase.properties • ID-based Database: <product_name>/<table_name>.sdkibdatabase.properties • Multi-Database: <product_name>/table <n>/<table_name>.sdktbdatabase.properties • Syslog: syslog/<product_name>.subagent.sdkrfilereader.properties • SNMP: <vendor>/sdksnmp.#.snmptrap.properties (where # = trap type) • Scanner (Normal Text): <vendor>.scanner.sdkrfilereader.properties • Scanner (XML): <vendor>.scanner. xqueryparser.properties • Scanner (Database): <vendor>. sdkdatabase.properties • REST: <product_name>.jsonparser.properties


Categorization (*.csv)

Development

Location under {$Connector_Home}/current • user/agent/acp/categorizer/current/<device_vendor>/<device_product>.csv

File format Getters and setters • Exact match, range, regex getters are all supported Single categorization file: vendor1/product1.csv Multiple categorization file: vendor1/product1.link.csv, vendor1/product.csv and vendor1/product.0.csv

Point to Note File name capitalization • All folder and file names are in small letters • Space or non-alphanumeric letters are converted into underscore ‘-’, such as

– AS/400 -> as_400 – Microsoft Windows -> microsoft_windows


Categorization (*.csv) (Continued)

Development

Point to Note (continued) • Try using Regex getter and put some catch all to enhance the categorization result, such as

– Logon.* (For all undefined event type with “Logon” prefix) – .* (For all other undefined event type)

• Required categorization fields regex.event.deviceEventClassId,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categoryDeviceType,set.event.categoryOutcome,set.event.categorySignificance,set.event.agentSeverity

• Obtain the ArcSight Categorization Technical Note


Key field assignment

Topics Time fields: • deviceReceiptTime, startTime, endTime

Device name for categorization link: • deviceVendor / deviceProduct

Event type: • name/deviceSeverity/deviceAction/deviceEventClassId

Custom fields • Use deviceCustom* and avoid flex* • flex* fields are reserved for console mapping to additional data

Source-destination vs attacker-target • Only use source/destination pair (e.g. sourceUserName/destinationUserName) • attacker/target pair: result of correlation, not to be assigned in FlexConnector

Assignable fields (list of assignable fields in FlexConnector Developer’s Guide)


Local testing (CSV/CEF output)

When I need this? To test and debug the parser/categorization in the development phase, it will be more efficient to verify locally rather than submitting to logger and ESM How to use it? • For connector destination, choose CEF log file or CSV log file • Select header row to help readablility • For CSV log file, you can choose the required fields as output


Packaging

For submission of FlexConnector for production usage, we recommend: • Single Package file • Folder structure following Connector installation standard • Put device vendor/device product/FlexConnector version number in the package name

Example Package: $deviceProduct_flexconnector_vx.x_yyyy-mm-dd.zip Content folder structure: acp/

fcp/

flexagent/


Documentation

Highly recommend to provide following documentation in the FlexConnector files for ongoing maintenance: • Parser comments • Configuration Documentation


Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3033 Speaker Sunny Suen

Please give me your feedback


Thank you

flexconnector deep dive and best practices name base event count detect time target address source...

Documents