©2013 BIZOSYS TECHNOLOGIES PRIVATE LIMITED

HSearch @ Intel Innovation Lab

© 2012 Bizosys Technologies Pvt Ltd.

The Use-case

Assessing Market Risk of an Investment Portfolio involving

15 billion calculations

Big Join in Hadoop3 million positions x 5000 risk models Each model consists 2M products.

To achieve =>6 months of historical data to be readily available while calculating risk.Current Status=> Only 5 days of prior data is immediately available, rest in archives.

Business Benefit

• Allows more broader time based risk assessment

• Solution avoids costly architectures such as in memory JVM cache based computing.

© 2012 Bizosys Technologies Pvt Ltd.

The Cluster

© 2012 Bizosys Technologies Pvt Ltd.

Sl. No.Descriptio

n

Machine Name

Machine 1 Machine 2 Machine 3 Machine 4

1 Platform S4600SDP S4600SDP S4600SDP S4600SDP

2Processor

Details

Xeon E5-4650, 2.7 GHz

Xeon E5-4650, 2.7 GHz

Xeon E5-4650, 2.7 GHz

Xeon E5-4650, 2.7 GHz

20M L3 cache, 8 Core

20M L3 cache, 8 Core

20M L3 cache, 8 Core

20M L3 cache, 8 Core

4 Memory16 x 8GB-PC3L-10600R

16 x 8GB-PC3L-10600R

16 x 8GB-PC3L-10600R

16 x 8GB-PC3L-10600R

5 Hard disk 300GB SAS 300GB SAS 300GB SAS 300GB SAS

6   250GB SSD 250GB SSD 250GB SSD238.5GB SSD (4 x 60GB SSD in LVM)

7 OS Details

Redhat Enterprise Linux 6.3 x64

Redhat Enterprise Linux 6.3 x64

Redhat Enterprise Linux 6.3 x64

Redhat Enterprise Linux 6.3 x64

/boot = 1GB /boot = 1GB /boot = 1GB /boot = 1GB

swap = 32GB swap = 32GB swap = 32GB swap = 32GB

/root = 100GB /root = 100GB /root = 100GB /root = 100GB

/data = 167GB /data = 167GB /data = 167GB /data = 167GB

/ssd = 250GB /ssd = 250GB /ssd = 250GB /ssd = 238.5GB

Infrastructure - Metals

© 2012 Bizosys Technologies Pvt Ltd.

1 Hadoop Hadoop 1.2

2 HSearch HSearch 0.94.4.41

3 JDK JDK 1.6.0_45

4 HDFS JDK Memory 4 GB

5 Hsearch JDK Memory 4 GB

Hadoop

© 2012 Bizosys Technologies Pvt Ltd.

Learning

First Run: 120Sec (No-Cache), 98Sec(Cache)

Setup1250 Models / Machine with 1 SSD /Machine.1 Hsearch instance/machine and max 64 threads/instance

Results120 Sec with OS Cache Disabled. 98 Sec with OS Cache Enabled.

ObservationHigh I/O wait and Low CPU usage.Software bottleneck with sequential I/O reads.

Action TakenCode modified to parallelized I/O reads

Second Run: 115Sec (No-Cache), 90Sec(Cache)

Setup1250 Models / Machine with 1 SSD /Machine.1 Hsearch instance/machine and max 64 threads/instance

Results115 Sec with OS Cache Disabled. 90 Sec with OS Cache Enabled.

ObservationAfter app log analysis we found DFSClient bottleneck.

Action TakenIntroduced 2 HSearch Instances/Machines

Third Run: 70 Sec (No-Cache), 34 Sec(Cache)

Setup1250 Models / Machine with 1 SSD /Machine.2 Hsearch instance/machine and max 32 threads/instance

Results70 Sec with OS Cache Disabled. 33.8 Sec with OS Cache Enabled.

Observation (No Cache)Average CPU Usage 32%, max 43%, Avg interrupt 17245 and avg context switch 6365 and avg I/O wait 9.16.

Action Taken4 SSD drives in a single machine.

Fourth Run: 32.8Sec (No-Cache) 30.3Sec(Cache)

Setup1250 Models / Machine with 2 instance/machine. 4 SSDs/Machine. Max 32 threads / instance40ms Delay on parallel thread launchResults32.8 Sec with OS Cache disabled. 30.3 Sec on cache enabled.

Observation (No Cache)Average CPU Usage 75%, max 97%, Avg interrupt 48921 and avg context switch 23376 and avg I/O wait 2.5.

Action TakenMore Delay is introduced to reduce contention.

Fifth Run: 32.5 Sec (No-Cache), 32Sec(Cache)

Setup1250 Models / Machine with 2 instance/machine. 4 SSDs/Machine. Max 32 threads / instance45ms Delay on parallel thread launchResults32.504 Sec with OS Cache Disabled. 32.060 Sec with OS Cache Enabled.

Observation (No Cache)Average CPU Usage 55%, max 82%, Avg interrupt 37564 and avg context switch 9419 and avg I/O wait 1.0.

Action TakenNone