ArcSight STIX/TAXII Python Client

The ArcSight STIX/TAXII Python Client is a Python package with command line scripts for downloading and processing collections from TAXII servers and convert the STIX data to CSV files.

Contents Chapter 1: Overview and Architecture .................................................................................................................................. 2

STIX and TAXII ..................................................................................................................................................................... 2

Active Threat Intelligence Package .................................................................................................................................... 2

How the ArcSight STIX/TAXII Python Client Works ........................................................................................................... 2

What ArcSight STIX/TAXII Python Client Can Do for You?................................................................................................ 3

Supported Platforms .......................................................................................................................................................... 3

Chapter 2: Installing the ArcSight STIX/TAXII Python Client ................................................................................................ 4

Requirements: .................................................................................................................................................................... 4

Installing the ArcSight STIX/TAXII Python Client ............................................................................................................... 6

Troubleshooting the Installation ....................................................................................................................................... 8

Chapter 3: Using the ArcSight STIX/TAXII Python Client ...................................................................................................... 8

Using the client from the Command Line .......................................................................................................................... 9

Basic Usage ..................................................................................................................................................................... 9

Using a configuration file ............................................................................................................................................. 12

Reading STIX XML Files ................................................................................................................................................. 15

Reading STIX files from us-cert.gov ............................................................................................................................. 15

Using the client as a CronJob ........................................................................................................................................... 16

Using the client without the ArcSight FlexConnector ..................................................................................................... 17

Basic Usage ................................................................................................................................................................... 17

Manually Import into ESM ........................................................................................................................................... 18

Chapter 4: Installing and Configuring the ArcSight FlexConnector .................................................................................... 21

Determine Which Configuration Is Needed .................................................................................................................... 21

Installing the ArcSight FlexConnector .............................................................................................................................. 21

Using the ArcSight SmartConnector GUI installer ....................................................................................................... 21

Using the ArcSight SmartConnector Console installer ................................................................................................ 28

Configuring the ArcSight FlexConnector .......................................................................................................................... 34

Chapter 5: Uninstall/Removing the ArcSight STIX/TAXII Python Client ............................................................................. 35

Uninstall/Removing the ArcSight STIX/TAXII Python Client ........................................................................................... 35

Chapter 1: Overview and Architecture

STIX and TAXII

STIX stands for Structured Threat Information eXpression and is a structured language for cyber threat intelligence.

TAXII stands for Trusted Automated eXchange of Indicator Information and is a free and open transport mechanism that standardizes the automated exchange of cyber threat intelligence.

When we talk about a STIX/TAXII server, it means that the server supports/understands the TAXII protocol for communication of cyber threat information in the STIX language.

https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=cti

Activate Threat Intelligence Package

The Activate Threat Intelligence Package is a package based on the Activate Framework that populates, displays and monitors the Threat Model, which is used to detect and contextualize potential malicious activity based on intelligence derived from a site-specific mix of threat intelligence sources.

For more information about the Activate Threat Intelligence Package see the foswiki

https://sec.microfocus.com/foswiki/bin/view/ArcSightActivate/L1ThreatIntelligence

How the ArcSight STIX/TAXII Python Client Works

The ArcSight STIX/TAXII Python Client is a set of Python scripts that uses the official TAXII/STIX/CYBOX modules for downloading collections from TAXII servers, and convert the data from a STIX format to a CSV file that is useable for the Activate Threat Intelligence Package.

Unstructured CTI

STIX/TAXII Server

Structured CTI

STIX/TAXII server

•Provides CTI

STIX/TAXII Script

•Downloads CTI and outputs to a CSV file.

FlexConnector

•Reads the CSV file using the properties file

What ArcSight STIX/TAXII Python Client Can Do for You?

ArcSight STIX/TAXII Python Client allows you to populate the Activate Threat Intelligence Threat Model automatically and manually with the Cyber Threat Intelligence from your trusted Threat Intelligence Sources.

Supported Platforms The package is written in the Python language which is supported on different platforms. See the Python website for more information about different platforms. Python 2.x: https://docs.python.org/2/using/index.html Python 3.x: https://docs.python.org/3/using/index.html

Chapter 2: Installing the ArcSight STIX/TAXII Python Client

Requirements:

1. Python version 2.7.x or 3.6.x 2. setuptools and pip (see below for installation instructions)

• Setuptools: Setuptools is a collection of enhancements to the Python distutils that allow developers to more easily build and distribute Python package

• Pip: pip is a package management system used to install and manage software packages written in Python • For installing setuptools and/or pip see https://packaging.python.org/tutorials/installing-packages/#install-

pip-setuptools-and-wheel

For reference, the following Python libraries are installed during the setup process:

• Python TAXII module (will be installed when running the setup) o TAXII: A Python library for handling TAXII Messages invoking TAXII Services. o https://github.com/TAXIIProject/libtaxi o https://libtaxii.readthedocs.io/

• Python TAXII2 client (will be installed when running the setup) o cti-taxii-client is a minimal client implementation for the TAXII 2.0 server o https://github.com/oasis-open/cti-taxii-client o https://taxii2client.readthedocs.io/en/latest/

• Python STIX module (will be installed when running the setup) o STIX: A Python library for parsing, manipulating, and generating STIX content. o https://github.com/STIXProject/python-stix o https://stix.readthedocs.io

• Python STIX2 module (will be installed when running the setup) o Python APIs for serializing and de- serializing STIX 2 JSON content o https://github.com/oasis-open/cti-python-stix2 o https://stix2.readthedocs.io/en/latest/

• Python STIX-EDH module (will be installed when running the setup) o stix-edh: An extension to python-stix supporting STIX Data Markings for the Enhance Shared Situational

Awareness (ESSA) Initiative’s Information Sharing Architecture (ISA) Access Control Specification (ACS), which are based on the US Intelligence Community’s Enterprise Data Header (EDH) specification.

• Python CybOX module (will be installed when installing STIX) o CybOX: A Python library for parsing, manipulating, and generating CybOX content o https://github.com/CybOXProject/python-cybox o https://cybox.readthedocs.io/

• Python Tabulate module (Optional, but will be installed when running the setup) o tabulate: for pretty printing (Optional) o https://pypi.python.org/pypi/tabulate

• Python Cmd2 module (will be installed when running the setup) o cmd2: for interactive commandline interface o https://pypi.python.org/pypi/cmd2

• Python Progressbar2 module (Optional, but will be installed when running the setup) o progressbar: text based progress o https://pypi.python.org/pypi/progressbar2 (Optional)

• Python pytz module (will be installed when running the setup) o pytz brings the Olson tz database into Python. This library allows accurate and cross platform timezone

calculations using Python 2.4 or higher. It also solves the issue of ambiguous times at the end of daylight saving time, which you can read more about in the Python Library Reference

o https://pypi.python.org/pypi/pytz • Python configparser module (will be installed when running the setup)

o This library brings the updated configparser from Python 3.5 to Python 2.6-3.5. o The ancient ConfigParser module available in the standard library 2.x has seen a major update in Python

3.2. This is a backport of those changes so that they can be used directly in Python 2.6 - 3.5. o https://pypi.python.org/pypi/configparser

• Python sqlalchemy module (will be installed when running the setup) o SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers

the full power and flexibility of SQL. SQLAlchemy provides a full suite of well known enterprise-level persistence patterns, designed for efficient and high-performing database access, adapted into a simple and Pythonic domain language.

o https://pypi.python.org/pypi/sqlalchemy • Python Requests module (will be installed when installing TAXII2-client)

o Requests is a HTTP library for Python o http://docs.python-requests.org/en/master/ o https://pypi.org/project/requests/

Installing the ArcSight STIX/TAXII Python Client

The ArcSight STIX/TAXII Python Client uses setuptools and can be installed with pip or easy_install Python only started bundling pip with Python 3.4. For earlier versions, pip needs to be “bootstrapped” Ensure you can run pip from the command line Make sure you can run Python (If you installed Python but cannot run Python, then you will need to check the PATH variable, on Windows the Python27\Scripts folder needs to be in the PATH variable also) Check if pip is installed by running the following command: pip --version For Python 3.x run the command: pip3 --version If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately. If pip isn’t installed, you can try to bootstrap it from the standard library: python -m ensurepip --default-pip For Python 3.x: python3 -m ensurepip --default-pip When the requirements are not satisfied, follow the instructions on the screen, your OS might allow you to install pip with the systems package manager. If this is not available you can manually download pip. Securely download the file get-pip.py from https://bootstrap.pypa.io/get-pip.py and run: python get-pip.py For Python 3.x run: python3 get-pip.py Note: Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. Before installing the package you must ensure that pip, setuptools and wheel are up to date: python -m pip install --upgrade pip setuptools wheel For Python 3.x:

Python3 -m pip install --upgrade pip setuptools wheel

Depending on your system’s configuration, you might need superuser/admin rights.

For more information visit https://packaging.python.org/tutorials/installing-packages/#ensure-you-can-run-pip-from-the-command-line

To install the package only for the current user, use the --user flag (without superuser/admin rights) Note: installing the package with the --user requires you to set the correct path for site packages. See https://pip.pypa.io/en/stable/user_guide/#user-installs and https://docs.python.org/2/library/site.html#site.USER_BASE for more information

Installing from local source

Python 2.7.x:

pip2 install arcsight_stix_taxii.zip

Or

easy_install-2.7 arcsight_stix_taxii.zip

Or (unzip arcsight_stix_taxii.zip)

cd arcsight_stix_taxii python2.7 setup.py install

Python 3.6.x:

pip3 install arcsight_stix_taxii.zip

Or

easy_install-3.6 arcsight_stix_taxii.zip

Or (unzip arcsight_stix_taxii.zip)

cd arcsight_stix_taxii python3.6 setup.py install

Note: Only pip is able to uninstall the package, with the other options you will need to delete the files manually.

Depending on your system’s configuration, the command python can be version 2.x or 3.x use python2 or python3 to make sure you run the correct version, same counts for pip, except for easy_install, this command requires the full version.

Validating the Install

To validate that install worked properly, run the command line client with the version option.

Upgrading the STIX/TAXII client

Python 2.7 pip2.7 install --upgrade arcsight_stix_taxii.zip easy_install-2.7 --upgrade arcsight_stix_taxii.zip Or (unzip arcsight_stix_taxii.zip)

cd arcsight_stix_taxii python2.7 setup.py install --force

Python 3.6 pip3.6 install arcsight_stix_taxii.zip easy_install-3.6 arcsight_stix_taxii.zip Or (unzip arcsight_stix_taxii.zip)

cd arcsight_stix_taxii python3.6 setup.py install --force

Troubleshooting the Installation

• Permission denied errors will happen when the user does not have the correct rights for installing, try installing the package with superuser/admin rights

• arcsight-taxii-client command not found. o ArcSight STIX TAXII client not installed properly or system variable PATH not correctly set

• arcsight-taxii-client not working after installation, ImportError: No module named arcsight_stix_taxii.client o ArcSight STIX TAXII client not installed properly, ensure pip, setuptools, and wheel are up to date. o https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-

date • warning: no files found matching '*.h' messages during installation, install python-devel package

Chapter 3: Using the ArcSight STIX/TAXII Python Client

Using the client from the Command Line Basic Usage STIX/TAXII v1 The v1 client has 2 required positional arguments, hostname and path (when using a TAXII server)

hostname : hostname TAXII server path : Path where the service is located

The main functions for interacting with the TAXII servers are:

--discover : Discover services on TAXII server --collection : Retrieve collections from TAXII server --poll : Poll collection (also known as feed)

arcsight-taxii-client hostname path --discover : Discover services on TAXII server arcsight-taxii-client hostname path --collection : Retrieve collections from TAXII server arcsight-taxii-client hostname path --poll collection : Poll collection The client has 4 optional arguments that does not require the positional arguments to be set.

--conf : Load configuration from file --sdkfilereader or -s : Write sdkfilereader.properties file --stix-file : Read from local STIX file --stix-folder : Read all STIX files in folder

Optional argument for --stix-folder

-r : Recursive

arcsight-taxii-client --conf configfile : Read from config file. arcsight-taxii-client --sdkfilereader name : Write sdkfilereader.properties file arcsight-taxii-client --stix-file : Read local STIX XML file arcsight-taxii-client --stix-folder : Read local STIX XML files in Folder

TAXII client specific argument

--auth : Set Authentication Type (Default = basic) --username : Set username --password : Set password (Will prompt for a password without echoing) --no-https : Required when TAXII server does not support HTTPS --port : Set port number, only necessary if port is not 80 or 443 (for port 80 use --no-https) --proxy : Set proxy, use system_proxy for system specified proxy

The client logs to a rotating log file in the current working directory. You can change the logging output folder with the --log argument. The optional arguments can be set with an = character or a space, for example: --username=guest or --username guest TAXII Content timestamp options:

--begin : Set exclusive begin date in ISO 8601 format (e.g 2017-12-15) --end : Set inclusive end date in ISO 8601 format (e.g 2017-12-15) --today : Use the current date at midnight

--days : Use the current date - x days --hours : Use the current date - x hours --months : Use the current date - x months

TAXII Content timestamp examples:

• --begin 2017-12-10 : Sets exclusive begin date to 2017-12-10 • --end 2017-12-15 : Sets inclusive end date to 2017-12-10 • --today : Sets exclusive begin date to current day at midnight • --days 7 : Sets exclusive begin date to current day - 7 days • --hours 24 : Sets exclusive begin date to current day - 24 hours • --months 6 : Sets exclusive begin date to current day - 6 months

Some TAXII servers might require a --begin and --end date for retrieving collections (e.g. in our testing, Anomali’s limo feed required these options).

When you want to poll a collection, you need to know what the service address is, some TAXII server will work with the same services address. To find out at what address the service are, you can use the --discover argument.

Discover services at hailataxii.com:

arcsight-taxii-client hailataxii.com /taxii-discovery-service --discover --no-https --auth basic --username guest

2017-11-13 18:36:36,889 : INFO : Discovering Services Address Type Available ----------------------------------- --------------------- ----------- http://hailataxii.com:80/taxii-data POLL True http://hailataxii.com:80/taxii-data COLLECTION_MANAGEMENT True http://hailataxii.com:80/taxii-data DISCOVERY True

Retrieve collections from hailataxii.com, the client will discover services first to obtain the correct address. It will show the collection name, description and if the collection is available

arcsight-taxii-client hailataxii.com /taxii-discovery-service --collection --no-https --auth basic --username guest

2017-11-13 18:37:51,871 : INFO : Discovering Services Address Type Available ----------------------------------- --------------------- ----------- http://hailataxii.com:80/taxii-data POLL True http://hailataxii.com:80/taxii-data COLLECTION_MANAGEMENT True http://hailataxii.com:80/taxii-data DISCOVERY True Collection Type Description Available -------------------------------- --------- -------------------------------- ----------- guest.dataForLast_7daysOnly DATA_FEED guest.dataForLast_7daysOnly True guest.EmergingThreats_rules DATA_FEED guest.EmergingThreats_rules True guest.phishtank_com DATA_FEED guest.phishtank_com True system.Default DATA_FEED system.Default True guest.EmergineThreats_rules DATA_FEED guest.EmergineThreats_rules True guest.dshield_BlockList DATA_FEED guest.dshield_BlockList True guest.Abuse_ch DATA_FEED guest.Abuse_ch True guest.MalwareDomainList_Hostlist DATA_FEED guest.MalwareDomainList_Hostlist True guest.Lehigh_edu DATA_FEED guest.Lehigh_edu True guest.CyberCrime_Tracker DATA_FEED guest.CyberCrime_Tracker True guest.blutmagie_de_torExits DATA_FEED guest.blutmagie_de_torExits True

Polling a collection at hailataxii.com requires also the correct path (address) and is case sensitive, you can use the correct path from discovering the service, or add the --auto argument and the client will discover the services and request the collections first (requesting the collection allows the client to check if the collection is available)

The argument --output is required when using --poll, --output specifies the output folder where the CSV needs to be written to, this folder needs to be the folder configured in the FlexConnector. For example: /home/user/stix_taxii

To poll multiple collections, you can use the --poll argument multiple times (--poll guest.Abuse_ch --poll guest.Lehigh_edu)

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --output /home/user/stix_taxii

2017-11-13 22:03:18,536 : INFO : Discovering Services 2017-11-13 22:03:20,332 : INFO : Requesting collection 2017-11-13 22:03:20,332 : INFO : Polling guest_Abuse_ch 2017-11-13 22:03:24,294 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out__uAyAM.csv

When the client is ready writing the CSV file, the FlexConnector will parse this file and send it to ESM Other optional arguments that can be used for polling are:

--cifv2 : Writes CSV in the format needed for using CIF FlexConnector --itype : Set the indicatorType --use-ttp : Use TTP Option or TTP Title as IndicatorType --producer : Set the Producer of the collection --score : Set score, default=50 --confidence : Set confidence, low, medium or high default=low --tlp-color : Set TLP color white, green, amber or red, default=white --group : Set group, default=everyone --relevance : Set relevance --reference : Set reference/altid --reference_tlp : Set reference/altid TLP color white ,green ,amber, or red, default=white --campaigns : Use short descriptions of campaigns as the description

Other optional arguments:

--us-cert : Loads us-cert specific STIX modules --active-list : Writes CSV in the active list format, needed when you want to do a manual import --log : Output folder for log files --memory : Store temporary data in memory instead of disk, might be a bit faster, but the client

will use more memory depending on the size of the collection. --keep-db : Keep database file after polling collection --db-file : Database file location for reading/writing --no-csv : Do not create CSV file, useful when you only want to write to the database file

--silent : Don't print output to screen, useful for using the client as a CronJob --debug : Show debug information

Basic Usage STIX/TAXII 2 The v2 client is called arcsight-taxii-client2

The main arguments for interacting with the TAXII2 servers are:

--url : URL of the TAXII2 server --collections : Retrieve collections from TAXII2 server --collection : Download collection

arcsight-taxii-client2 --url <url> --collections : Retrieve collections from TAXII2 server arcsight-taxii-client2 --url <url> --collection : Download collection from TAXII2 server

The client has 4 optional arguments that does not require the positional arguments to be set.

--conf : Load configuration from file --sdkfilereader or -s : Write sdkfilereader.properties file --stix-file : Read from local STIX file --stix-folder : Read all STIX files in folder

Optional argument for --stix-folder

-r : Recursive

arcsight-taxii-client2 --conf configfile : Read from config file. arcsight-taxii-client2 --sdkfilereader name : Write sdkfilereader.properties file arcsight-taxii-client2 --stix-file : Read local STIX2 JSON file arcsight-taxii-client2 --stix-folder : Read local STIX2 JSON files in Folder

TAXII client specific argument

--auth : Set Authentication Type (Default = basic) --username : Set username --password : Set password (Will prompt for a password without echoing) -- no-verification : Don't validate the entity credentials --proxy : Set proxy schema;url for example --proxy “http;http:10.0.0.1:8080”

The client logs to a rotating log file in the current working directory. You can change the logging output folder with the --log argument. The optional arguments can be set with an = character or a space, for example: --username=guest or --username guest Content timestamp options:

--begin : Set exclusive begin date in ISO 8601 format (e.g 2017-12-15) --end : Set inclusive end date in ISO 8601 format (e.g 2017-12-15) --today : Use the current date at midnight --days : Use the current date - x days --hours : Use the current date - x hours --months : Use the current date - x months --valid-until : Use the above date argument as valid-until

Content timestamp examples:

• --begin 2017-12-10 : Sets exclusive begin date to 2017-12-10 • --end 2017-12-15 : Sets inclusive end date to 2017-12-10

• --today : Sets exclusive begin date to current day at midnight • --days 7 : Sets exclusive begin date to current day - 7 days • --hours 24 : Sets exclusive begin date to current day - 24 hours • --months 6 : Sets exclusive begin date to current day - 6 months

Retrieve collections from TAXII2 server. The client will show the collection name, description and if the collection is readable arcsight-taxii-client2 --url http://127.0.0.1:5000/taxii/ --username user1 -p --collections Password: Collection Description Readable --------------- ----------------------------------------- ---------- Test Indicators This data collection is for test purposes True

Collection names are case sensitive, collection names with spaces needs enclosed with quotes, and must be readable.

The argument --output is required when using --collection, --output specifies the output folder where the CSV needs to be written to, this folder needs to be the folder configured in the FlexConnector. For example: /home/user/stix_taxii

To poll multiple collections, you can use the --collection argument multiple times (--collection collection1 --collection collection2)

arcsight-taxii-client2 --url http://127.0.0.1:5000/taxii/ --username user1 -p --collection "Test Indicators " --output /home/user/stix_taxii

2018-12-05 14:38:52,219 : INFO : Downloading collection: Test Indicators 2018-12-05 14:38:52,234 : INFO : Writing data to /home/user/stix_taxii/out_4wsejidu.csv

When the client is ready writing the CSV file, the FlexConnector will parse this file and send it to ESM Other optional arguments that can be used for downloading a collection are:

--cifv2 : Writes CSV in the format needed for using CIF FlexConnector --itype : Set the indicatorType --producer : Set the Producer of the collection --score : Set score, default=50 --confidence : Set confidence, low, medium or high default=low --tlp-color : Set TLP color white, green, amber or red, default=white --group : Set group, default=everyone --relevance : Set relevance --reference : Set reference/altid --reference_tlp : Set reference/altid TLP color white ,green ,amber, or red, default=white --campaigns : Use short descriptions of campaigns as the description --active-list : Writes CSV in the active list format, needed when you want to do a manual import --log : Output folder for log files --silent : Don't print output to screen, useful for using the client as a CronJob --debug : Show debug information

Using a configuration file All the above arguments can be defined in a configuration file, use the argument --config configfile.conf

The argument --config can be used in combination with --stix-file, without the argument --stix-file the client will poll the collection specified in the configuration file, with the --stix-file argument the client will read the STIX file specify in the argument. Note: some configuration options are different when using the arcsight-taxii-client2 command.

Example configuration file for polling guest.Abuse_ch hailataxii.com using the default stixtaxii CSV layout

[app] us-cert = False memory = False cleanup = True [server] hostname = hailataxii.com port = None path = /taxii-discovery-service https = False auth_type = basic username = guest password = changeme collections = guest.Abuse_ch auto = True proxy = None begin_date = None end_date = None [csv] order = otype,observable,itype,firstdetecttime,lastdetecttime,score,confidence,source,relevance,description_or_title,reference datetime_format = %Y-%m-%d %H:%M:%S %z output = /home/user/stix_taxii itype = suspicious ttp_option = type score = 50 confidence = low tlp_color = white group = everyone relevance = reference = reference_tlp = white activelist = False cifv2 = False campaigns = False

To create a configuration file from the current command line arguments, use --create-config filename.conf

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --output /home/user/stix_taxii --create-config hailataxii.conf

2017-11-13 22:03:30,484 : INFO : Writing configuration file to: hailataxii.conf Polling hailataxii.com using configuration file arcsight-taxii-client --conf hailataxii.conf 2017-11-13 22:03:18,536 : INFO : Using configuration file 2017-11-13 22:03:18,536 : INFO : Discovering Services 2017-11-13 22:03:20,332 : INFO : Requesting collection 2017-11-13 22:03:20,332 : INFO : Polling guest_Abuse_ch 2017-11-13 22:03:24,294 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out__uAyAM.csv

Reading STIX Files

The client can read STIX v1 XML and STIX v2 JSON files using the argument --stix-file stix-file-name, filenames that contains spaces must be enclosed with quotes, the --stix-file argument can be used multiple times.

For reading STIX v2 JSON files, you need to use the arcsight-taxii-client2 command

arcsight-taxii-client --stix-file=”STIX File.xml” --output /home/user/stix_taxii/ 2017-11-13 22:04:18,537 : INFO : Reading STIX file 2017-11-13 22:04:18,867 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out_CasFaA.csv

Reading US-CERT STIX XML file using a configuration file (Only possible with the v1 client) arcsight-taxii-client --conf uscert.conf --stix-file=”STIX File.xml” 2017-11-13 22:04:18,536 : INFO : Using configuration file 2017-11-13 22:04:18,537 : INFO : Reading STIX file 2017-11-13 22:04:18,867 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out_CasFaA.csv

The client can read all STIX files in a folder by using the argument --stix-folder, this argument can also be used multiple times. (use -r for recursive)

Reading STIX v1 files from us-cert.gov

us-cert.gov (United States Computer Emergency Readiness Team) provides downloadable critical cybersecurity information on their website in CSV and STIX format.

For example the North Korean Malicious Cyber Activity: https://www.us-cert.gov/hiddencobra IOC’s: https://www.us-cert.gov/ncas/alerts/TA17-318B Reading STIX files from US-CERT requires the client to load extra STIX modules, this can be done by using the --us-cert argument. (Only possible with the v1 client)

Reading US-CERT STIX XML File

arcsight-taxii-client --stix-file=” TA-17-318B-IOCs.xml” --output /home/user/stix_taxii/ --us-cert 2017-11-13 22:04:18,536 : INFO : Registering CISCP namespace 2017-11-13 22:04:18,537 : INFO : Reading STIX file 2017-11-13 22:04:18,867 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out_CasFaA.csv Reading US-CERT STIX XML file using a configuration file with us-cert enabled arcsight-taxii-client --conf=uscert.conf --stix-file=”TA-17-318B-IOCs.xml” 2017-11-13 22:04:18,535 : INFO : Using configuration file 2017-11-13 22:04:18,536 : INFO : Registering CISCP namespace 2017-11-13 22:04:18,537 : INFO : Reading STIX file 2017-11-13 22:04:18,867 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing data to /home/user/stix_taxii/out_CasFaA.csv

Using the client as a CronJob

All client arguments can be used in a CronJob, recommended is to the --silent and --log arguments, using these arguments the client won’t print to the screen, but will keep logging to a log file, with the --log argument you can specify the folder where the log file should be stored.

Example CronJob for polling guest.Abuse from hailataxii.com

This example for the STIX/TAXII v1 client will run the client every day at 00:00 and poll guest.Abuse_ch from hailataxii.com (path to arcsight-taxii-client might differ) and requests TAXII content from the last 24 hours

Run crontab -e (depending on your system’s configuration, you might need to run crontab -e as a superuser) and add the following line:

0 0 * * * /usr/local/bin/arcsight-taxii-client hailataxii.com /taxii-data --poll guest.Abuse_ch --no-https --auth basic --username guest --output /home/user/stix_taxii/ --log /home/user/logs --silent --hours 24

Note: You can also use the --auto function

Using a configuration file to poll guest.Abuse_ch from hailataxii.com

0 0 * * * /usr/local/bin/arcsight-taxii-client --conf hailataxii.conf --log /home/user/logs --silent --hours 24

Not: Not all argument shown in the example above work for the v2 client.

Using the client without the ArcSight FlexConnector

Basic Usage Instead of writing CSV files for the FlexConnector, the client is also able to write 2 CSV files, for manual import into the Active Threat Intelligence Package.

The client writes a file for the suspicious addresses list, and a file for the suspicious entities list.

Polling guest.Abuse_ch from hailataxii.com and output to the active list format

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --output /home/user/stix_taxii --activelist

2017-11-13 22:03:18,536 : INFO : Discovering Services 2017-11-13 22:03:20,332 : INFO : Requesting collection 2017-11-13 22:03:20,332 : INFO : Polling guest_Abuse_ch 2017-11-13 22:03:24,294 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing CSV in Activate Threat Intelligence Active List Format 2017-11-13 22:03:30,485 : INFO : Writing data to /home/user/stix_taxii/suspicious_addresses_O2Y6dl.csv 2017-11-13 22:03:30,486 : INFO : Writing data to /home/user/stix_taxii/suspicious_entities_x1dE8T.csv

Using a configuration file to poll guest.Abuse_ch from hailataxii.com and output to the active list format (set activelist = True in the configuration file)

arcsight-taxii-client --conf hailataxii.conf 2017-11-13 22:03:18,536 : INFO : Using configuration file 2017-11-13 22:03:18,536 : INFO : Discovering Services 2017-11-13 22:03:20,332 : INFO : Requesting collection 2017-11-13 22:03:20,332 : INFO : Polling guest_Abuse_ch 2017-11-13 22:03:24,294 : INFO : Storing temporary data on disk 2017-11-13 22:03:30,484 : INFO : Writing CSV in Activate Threat Intelligence Active List Format 2017-11-13 22:03:30,485 : INFO : Writing data to /home/user/stix_taxii/suspicious_addresses_O2Y6dl.csv 2017-11-13 22:03:30,486 : INFO : Writing data to /home/user/stix_taxii/suspicious_entities_x1dE8T.csv

The above commands will request the everything from the collection, use the arguments --begin, --end, --today, --days or --hours for requesting TAXII content within a specific time range.

Manually Import into ESM At times you may want to manually bring new CSV files into the Activate Threat Intelligence Active Lists. Please note that this is not required when the CSV is being processed by the Flex Connector.

• Navigate to Active Lists/Shared/ArcSight Activate/Solutions/Threat Intelligence/Indicators and Warnings.

• Right click on Suspicious Addresses List and click on Import CSV File…

• Select the correct CSV file, for the Suspicious Addresses list select the suspicious_addresses_*.csv file and click on Open

• Verify the data with the Import Preview and click on Import

• Verify if the data is imported into the Suspicious Addresses List, right click on Suspicious Addresses List and click on Show Entries.

After importing the suspicious_addresses_*.csv file, the Suspicious Addresses List should be filled with the guest.Abuse_ch IOC’s

• Repeat the above steps for the Suspicious Entities List. • The suspicious_entities_*.csv files needs to be imported into the Suspicious Entities List.

After importing suspicious_entities_*.csv files, the Suspicious Entities List should be filled with the guest.Abuse_ch IOC’s

Chapter 4: Installing and Configuring the ArcSight FlexConnector

Determine Which Configuration Is Needed The ArcSight STIX/TAXII Python Client is shipped with 2 different CSV formats for use with the ArcSight FlexConnector.

If you are already using CIF and/or the ArcSight FlexConnector with the cifv2.sdkfilereader.properties configuration file, you can skip installing the ArcSight FlexConnector and you need to use the CIFv2 CSV format.

Read chapter 3, for instruction on how to use the ArcSight STIX/TAXII Python Client with the CIFv2 CSV format.

If you don’t want to use CIF or the cifv2.sdkfilereader.properties configuration file, you can use the default stixtaxii sdkfilereader.properties configuration file, follow the instructions at “Installing the ArcSight FlexConnector“

Installing the ArcSight FlexConnector In order to Import the Cyber Threat Intelligence into the Active Threat Intelligence Package automatically you will need to install the ArcSight FlexConnector.

The ArcSight FlexConnector normalizes the output generated from the ArcSight STIX/TAXII Python Client.

Settings that needs to be set:

- Connector Type: FlexConnector Multiple Folder File - Folder: The folder that needs to be monitored for CSV files - Processing Mode: batch - Configuration File: File matching the CSV Format (stixtaxii or cifv2) - Configuration Type: sdkfilereader

Using the ArcSight SmartConnector GUI installer Start the ArcSight SmartConnector Installer and follow the instructions.

You may get an "unsupported platform message, which you can safely ignore in this case.

Click Next

Choose the installation folder, and click Next

Choose the folder where you want to create links, it is save to choose ‘Don’t create links’ and click Next

Verify the information and click on Install

Select Add a Connector and click on Next

Select type “ArcSight FlexConnector Multiple Folder File” and click on Next

Select “false” and click on Next

Click on Add, and set the correct values and click on Next

For example:

Folder: The folder where you want to store the CSV files generated by the ArcSight STIX/TAXII Python Client Processing Mode: Batch Configuration File: stixtaxii (cifv2 for cifv2 CSV layout) Configuration Type: sdkfilereader

Select destination type and click on Next

Enter destination parameters and click on Next

Enter the connector details and click on Next

Select “Import the certificate to connector from destination” and click on Next

Verify information and click on Next

Follow the instructions if you want the FlexConnector to run as a service. Click on Next

Choose ‘Exit’ to quit the installer and click on Next

Click on Done

Proceed to “Configuring the ArcSight FlexConnector“

Using the ArcSight SmartConnector Console installer For the purposes of this install, we used ArcSight -7.4.0.7963.0-Connector_Linux64.bin

You may get an "unsupported platform message”, which you can safely ignore in this case.

Start the ArcSight SmartConnector Installer and follow the instructions.

Parts that needs attention is marked Yellow

user@arcsight:~$ ./ArcSight-7.4.0.7963.0-Connector-Linux64.bin Preparing to install... Extracting the JRE from the installer archive... Unpacking the JRE... Extracting the installation resources from the installer archive... Configuring the installer for this system's environment... Launching installer... =============================================================================== ArcSight SmartConnector (created with InstallAnywhere) ------------------------------------------------------------------------------- Preparing CONSOLE Mode Installation... =============================================================================== Platform Verification --------------------- You are installing this product on an unsupported platform. Please refer to the ArcSight SmartConnector Product and Platform Support Config document to find out about platforms supported for this release. To cancel this installation, click [Cancel]. ->1- OK 2- Cancel ENTER THE NUMBER OF THE DESIRED CHOICE, OR PRESS <ENTER> TO ACCEPT THE DEFAULT: 1 =============================================================================== Introduction ------------ The ArcSight Installer will guide you through the installation of the ArcSight SmartConnector. The first step installs the core ArcSight SmartConnector components; then you select the ArcSight SmartConnector you wish to configure. ArcSight recommends that you quit all other programs before continuing with this installation. Respond to each prompt to proceed to the next step in the installation. If you want to change something on a previous step, type 'back'. To cancel this installation at any time, type 'quit'. PRESS <ENTER> TO CONTINUE: =============================================================================== Choose Install Folder --------------------- Choose the folder where you would like to install an ArcSight SmartConnector. It is strongly recommended that you choose the folder name according to the device that you want to connect to, for example /ciscoids or /checkpointng. If you are upgrading an ArcSight SmartConnector from a previous version, please select the folder where the ArcSight SmartConnector is currently installed. Where would you like to install? Default Install Folder: /home/user/ArcSightSmartConnectors ENTER AN ABSOLUTE PATH, OR PRESS <ENTER> TO ACCEPT THE DEFAULT

: /home/user/ArcSightSmartConnectors INSTALL FOLDER IS: /home/user/ArcSightSmartConnectors IS THIS CORRECT? (Y/N): y =============================================================================== Choose Link Location -------------------- Where would you like to create links? ->1- Default: /home/user 2- In your home folder 3- Choose another location... 4- Don't create links ENTER THE NUMBER OF AN OPTION ABOVE, OR PRESS <ENTER> TO ACCEPT THE DEFAULT : 4 =============================================================================== Pre-Installation Summary ------------------------ Please Review the Following Information Before Continuing: Product Name: ArcSight SmartConnector Install Folder: /home/user/ArcSightSmartConnectors Link Folder: DO NOT INSTALL PRESS <ENTER> TO CONTINUE: =============================================================================== Installing... ------------- [==================|==================|==================|==================] [------------------|------------------|------------------|------------------] =============================================================================== Installation Complete --------------------- The core components of the ArcSight SmartConnector have been successfully installed to: /home/user/ArcSightSmartConnectors To finish the configuration of the SmartAgent, please go to the folder: /home/user/ArcSightSmartConnectors/current/bin/ and execute the script: ./runagentsetup.sh PRESS <ENTER> TO EXIT THE INSTALLER: user@arcsight:~$ cd /home/user/ArcSightSmartConnectors/current/bin/ user@arcsight:~/ArcSightSmartConnectors/current/bin$ ./runagentsetup.sh Assuming ARCSIGHT_HOME: /home/user/ArcSightSmartConnectors/current Assuming JAVA_HOME: /home/user/ArcSightSmartConnectors/current/jre ArcSight Agent Setup starting... Connector Setup Wizard starting in mode [CONSOLE]

[Thu Nov 23 14:23:18 CET 2017] [INFO ] Checking for a running instance of connector... [Thu Nov 23 14:23:19 CET 2017] [INFO ] Starting up connector... Connector Setup --------------- -------------------------------------------------------------------------------- What would you like to do? 0- Add a Connector 1- Set Global Parameters Please select an option: [Add a Connector] [0..1/cancel] :0 -------------------------------------------------------------------------------- Select the connector to configure Type: 0- Amazon Web Services CloudTrail 1- Apache HTTP Server Access File 2- Apache HTTP Server Error File 3- Apache Tomcat File 4- ArcSight Asset Import File 5- ArcSight CEF Cisco FireSIGHT Syslog 6- ArcSight CEF Encrypted Syslog (UDP) 7- ArcSight Common Event Format File 8- ArcSight Common Event Format Hadoop 9- ArcSight Common Event Format Multiple File 10- ArcSight Common Event Format REST 11- ArcSight FlexConnector CounterACT 12- ArcSight FlexConnector File 13- ArcSight FlexConnector ID-Based DB 14- ArcSight FlexConnector JSON Folder Follower 15- ArcSight FlexConnector Multiple DB 16- ArcSight FlexConnector Multiple Folder File 17- ArcSight FlexConnector Regex File 18- ArcSight FlexConnector Regex Folder File 19- ArcSight FlexConnector REST (N)ext - ------------- Next page ------------- Please select an option [0..19]: 16 Please verify the following parameters Type: ArcSight FlexConnector Multiple Folder File Are the values correct [yes/no/back/cancel]?yes -------------------------------------------------------------------------------- Enter the parameter details Log Unparsed Events?: 0- true 1- false Please select an option [0..1][false]: 1 Please verify the following parameters Log Unparsed Events?: false Are the values correct [yes/no/back/cancel]?yes | | 0%Verifying the parameters |########################################| 100% -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Enter the device details -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Row#|Folder|Processing Mode|Configuration File|Configuration Type -------------------------------------------------------------------------------- Please select an option: [A]dd [D]elete [I]mport [E]xport [F]inish =>a add

Folder[]: /home/user/stix_taxii The folder where you want to store the CSV files generated by the ArcSight STIX/TAXII Python Client Processing Mode: 0- batch 1- realtime Please select an option [0..1][batch]: 0 Configuration File[]: stixtaxii The configuration file you want to use, stixtaxii (cifv2 for cifv2 CSV layout) Configuration Type: 0- sdkfilereader 1- sdkrfilereader 2- sdkkeyvaluefilereader 3- cef Please select an option [0..3][sdkrfilereader]: 0 -------------------------------------------------------------------------------- Enter the device details -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Row#|Folder |Processing Mode|Configuration File|Configuration Type -------------------------------------------------------------------------------- 0 |/home/user/stix_taxii|batch |stixtaxii |sdkfilereader should look like this example -------------------------------------------------------------------------------- Please select an option: [A]dd [D]elete [I]mport [E]xport [F]inish =>f Finish Are the values correct [yes/no/back/cancel]?yes | | 0%Verifying the parameters |########################################| 100% -------------------------------------------------------------------------------- Enter the type of destination 0- ArcSight Manager (encrypted) 1- ArcSight Logger SmartMessage (encrypted) 2- ArcSight Logger SmartMessage Pool (encrypted) 3- CEF File 4- Event Broker (CEF Kafka) 5- CEF Syslog 6- CEF Encrypted Syslog (UDP) 7- CSV File 8- Raw Syslog Please select an option: [ArcSight Manager (encrypted)] [0..8/back/cancel] :0 -------------------------------------------------------------------------------- Enter the destination parameters *************************************************************** WARNING: Some of the required parameters will contain security sensitive information. Do you want to hide the input for these parameters from the screen?[yes/no] (typically you would answer 'NO' only if you are using a slow link (like a serial RS232 or a very slow network link) since this may add additional delays to the connection. If you are not sure, then select 'YES' or hit enter. *************************************************************** [yes]?yes Input for private parameters will be hidden. Manager Hostname: vm-esm611-demo Manager Port[8443]: 8443 User: admin Password: AUP Master Destination:

0- true 1- false Please select an option [0..1][false]: 1 Filter Out All Events: 0- true 1- false Please select an option [0..1][false]: 1 Enable Demo CA: 0- true 1- false Please select an option [0..1][false]: 1 Please verify the following parameters Manager Hostname: vm-esm611-demo Manager Port: 8443 User: admin Password: ******** AUP Master Destination: false Filter Out All Events: false Enable Demo CA: false Are the values correct [yes/no/back/cancel]?yes -------------------------------------------------------------------------------- Enter the connector details Name[]: STIX/TAXII Connector Location[]: Lab DeviceLocation[]: Lab Comment[]: STIX/TAXII Connector To Populate Threat Model Please verify the following parameters Name: STIX/TAXII Connector Location: Lab DeviceLocation: Lab Comment: STIX/TAXII Connector To Populate Threat Model Are the values correct [yes/no/back/cancel]?yes Registering destination |########################################| 100% -------------------------------------------------------------------------------- Following certificate will be imported into connector trust store: Host/port: vm-esm611-demo_8443 Details: CN=vm-esm611-demo, OU=ESP, O=HPESP, L=Sunnyvale, ST=CA, C=US 0- Import the certificate to connector from destination 1- Do not import the certificate to connector from destination Please select an option: [Import the certificate to connector from destination] [0..1/back/cancel] :0 | | 0%Importing certificate, registering destination and restarting the container |########################################| 100% -------------------------------------------------------------------------------- Add connector Summary Following are the added connector details: Connector Name [STIX/TAXII Connector], Connector Type [sdkmultifolderreader] Must be sdkmultifolderreader Continue [yes] ?yes

To run the FlexConnector as a service, follow these instructions -------------------------------------------------------------------------------- To modify service configuration, you must be running as root To manually configure, log on as root and execute the following script: /home/user/ArcSightSmartConnectors/current/bin/arcsight agentsvc -i -u user To manually remove the service, log on as root and execute the following script: /home/user/ArcSightSmartConnectors/current/bin/arcsight agentsvc -r Continue [yes] [yes/no/back/cancel]?yes -------------------------------------------------------------------------------- Would you like to continue or exit? 0- Continue 1- Exit Please select an option: [Continue] [0..1/back/cancel] :1 [Thu Nov 23 14:30:33 CET 2017] [INFO ] Shutting Down Agent Framework Version [7.4.0.7963.0] Shutting down Agent Modules now... Shutting down Agent Setup Wizard...done.

Configuring the ArcSight FlexConnector

The FlexConnector will try to process any files in the configured folder (in this example this is /home/user/stix_taxii)

To narrow down the files it uses input and thus avoid errors if anything inadvertent gets placed in this directory, we suggest changing the following parameter in <ArcSightSmartConnector>/current/user/agent/agent.properties (in this example this is /home/user/ArcSightSmartConnectors/current/user/agent/agent.properties) from:

agents[0].foldertable[0].wildcard=*.*

to:

agents[0].foldertable[0].wildcard=*.csv

The FlexConnector renames files in the configured folder (in this example the folder is /home/user/stix_taxii) after it processes them by appending ".processed" to the filename. To change this, once again edit agent.properties and change the following parameter from:

agents[0].foldertable[0].mode=RenameFileInTheSameDirectory

to:

agents[0].foldertable[0].mode=Delete

After configuring the ArcSight FlexConnector, the sdkfilereader.properties file needs to be copied to the ArcSight FlexConnector, the ArcSight STIX/TAXII Python Package contains 2 sdkfilereader.properties files and can be copied to the ArcSight FlexConnector folder using the client.

arcsight-taxii-client -s stixtaxii --out <ArcSightSmartConnector>/current/user/agent/flexagent

Where <ArcSightSmartConnector> is the folder where the FlexConnector is located, for example /home/user/ArcSightSmartConnectors

To copy the cifv2 sdkfilereader.properties file, use -s cifv2

After the coping the sdkfilereader.properties file, the ArcSight FlexConnector can be started.

Chapter 5: Uninstall/Removing the ArcSight STIX/TAXII Python Client

Uninstall/Removing the ArcSight STIX/TAXII Python Client

To uninstall the ArcSight STIX TAXII Package depends on how you installed the package, if you used pip, then you can uninstall the package with pip uninstall arcsight_stix_taxii, otherwise you will need to remove the package manualy.

The location of the Python package depends on what OS you use. For example on Debian/Ubuntu the Python2.7 package is located in /usr/local/lib/python2.7/dist-packages/ and is called arcsight_stix_taxii-1.1-py2.7.egg (1.1 is the current version at the time of writing)

On Windows, if Python2.7 is installed in C:\Python27\Lib\site-packages\

The installer also creates three scripts/launchers, on non-windows like Debian/Ubuntu the location of these commandline scripts is /usr/local/bin/ and the files are called arcsight-taxii-client, arcsight-taxii-client2 and arcsight-taxii-cli

On Windows the executables are located in C:\Python27\Scripts\ and the files are called arcsight-taxii-client.exe, arcsight-taxii-client2.exe, arcsight-taxii-client.exe.manifest, arcsight-taxii-client2.exe.manifest , arcsight-taxii-cli.exe and arcsight-taxii-cli.exe.manifest