grid computing and trade analytics with elastic - jay chin's presentation at elasticontour 2015

Jay Chin – jay.chin@excelian.com

Principal Consultant, Excelian

3 November 2015

0

Grid Computing and Trade Analytics

with Elastic

Excelian Technical Consulting – Who we are

1

Financial Services specialists

Distributed computing specialists since 2006

Experts in niche and emerging technologies

Financial Services – Insatiable appetite for Compute

• Algorithms (Computers) that actually do the trading - Remember the Flash Crash of May 6 2010? This is a result of HFT stopping and not trading causing the Market to drop 6% in mere minutes.

• Financial modelling - Use complex mathematical models to deal with asset prices, market movement, portfolio returns, etc.

• Huge amounts of data to process - e.g. connecting to one of the exchanges FIX, there will be up to 100,000 messages to process per second.

2

Source: Information Week, Wall Street & Technology Source: The Telegraph

What do compute grids look like ?

3

Typical Numbers For A Standard Grid

- 40k cores/engines

- 30m tasks

- 120 GB of Log metrics

- 60 – 80% Average Utilisation

- Data retention up to 6 Months

https://flic.kr/p/ydnEvw

Grid Maturity in Financial Services

4

HPC Maturity

Benchmark

2014

Tier I = Tier I banks

Tier II = Tier II banks

Point = point solutions used only for a specific use case (e.g. behind a software package, only for one business line…)

Mat

uri

ty L

evel

It is fairly common for bank to have grids. Larger banks tend to have at least 30,000 cores.

Case Study: ELK for Enterprise Grid Reporting Framework

Requirements:

• Enterprise Grid with 40,000 Cores across 4 Data centers in 2 Countries

• Reporting Dashboard for Grid Metrics

• Scalable up to 100,000 cores and 200 million Grid tasks per day

5

Goal: Architect an Enterprise Grid and design a Grid metrics reporting framework

for a top-tier investment bank.

The Case for ELK

6

Features ElasticSearchIntuitive Interface

Ease of Use

Security Integration

Scalability

Support

Pricing

Features

Integration with Grid Middleware

Elasticsearch met all the requirements except for the last one,which required some work on our part.

Initial Architecture: Single cluster across 2 regions

7

curl -XPUT localhost:9200/GridA_metrics/_settings -d '{"index.routing.allocation.include.tag" : “region_A" }'

Architecture After Consultation with Elastic Platinum Support

8

Challenges

9

• Bespoke deployment due to security restrictions in Bank’s Datacentre

• https://github.com/Excelian/ansible_fs_elkstack

• Development of custom ETL to query Grid Metrics database and load them into ElasticSearch

https://flic.kr/p/eqJHbr

More ELK Goodness

• Bank was very impressed with the reporting capabilities

• Support team at Elastic was also superior compared to some of the big vendors we were dealing with

10

AS A RESULT

1. We were tasked to do log centralization using Logstash

2. Explore Watcher for monitoring Grid and applications

Feedback from Investment Bank

• For the first time ever, developers were able to view Grid metrics and correlate them with logging events from a single interface

• Application teams are experimenting with Elastic

• Developers rethinking logging

11

Key Takeaways

• Lots of opportunities and interest in ElasticSearch in Financial services

• Single tool to do log analytics, alerting, events, searching, and metrics

• Elastic ticks all the right boxes for financial services: Security, scalability, support SLAs, etc.

• Elastic Platinum support has been fantastic

• Advanced Use Cases : Fraud Detection, Trade surveillance, Market Sentiment Analysis

12

13

Thank you!

If you have any feedback, please get in touch:

jay.chin@excelian.com

If you would like to join our community of technologists at Excelian please have a look at our careers page for the latest vacancies:

www.excelian.com/careers/

@Excelian

@Excelian

@ExcelianLTD

www.elastic.co

14