fast big data ingest into sap hana
TRANSCRIPT
1confidential
Fast Big Data Ingestinto SAP HANA
Denis KingSeptember, 2016
2confidential
Speaker Introductiono SVP Field Operations, Solace Inc.
o Working @ Solace for 12 years
o Many years working in Capital Markets, Telco and Government industries
o Focus mainly on networking and middleware
3confidential
Enterprise IT circa 1990
Application#1
Application#n
Imagine……achieving HA
…application overhead…Apps going offline
…spanning datacentres
COMPLEX & !SCALE
4confidential
Enter the Enterprise Bus
Application#1
Application#n
Connect once, the bus handles everything
5confidential
Big Data, the Apache way…
“How can we get all of that data into HANA/Hadoop?”
6confidential
Big Data Ingestion the Apache + Kafka way…
API
API
API
API
API
API
API
API
API
“How can we get all of that data into Hadoop?”
7confidential
Big Data Team Enterprise Architecture Team
8confidential
Open Source (Rabbit) Kafka0
50
100
150
200
250
300
350
400
450
Message Broker vs Kafka Throughput
Open Source JMS Kafka Solace VMR0
50
100
150
200
250
300
350
400
450
500
Open Source (Rabbit) Kafka Solace VMR Solace 35600
500
1000
1500
2000
2500
3000
3500
4000
o 1 server‐ 1 message broker vs
10 Kafka flows
o 1K messageso Java client APIs
Test configurations:
K m
essa
ges/
sec
9confidential
Big Data – The Simpler Approach
“Subscribe”
10CONFIDENTIAL
Big Data Lake MEET Big Data RIVER
11confidential
Big Data Lake meet Big Data RIVER
BigDataLake
Enterprise Big Data River
AMQP
JMS
MQTT
REST
JMSAMQP
12CONFIDENTIAL
So…what does an Enterprise Big Data RIVER need?
13confidential
Multi-Protocol, Multi QoS, Multi-pattern1234
14confidential
OpenPub/Sub
Req/ReplyWeb/Streaming
RESTfulWAN
JMSAMQPMQTTREST
PersistentNon-
persistentLow
LatencyHigh/Low Volume
Data Movement
Linking applications, devices and people across
any cloud, any platform, anywhere around the world.
Any Protocol Any QoS Any Pattern
15confidential
Event Driven, across datacenters, private, public clouds. Universal fabric
1234
1
16confidential
Distributed Big Data Rivero Large enterprise moving
workloads to the cloud, refreshing IT strategy
o Running workloads across public cloud, private cloud and on-premise systems
o Big Data lake subscribes to any data from any source
PublicClouds
On Premise
Private Cloud
App App App
App App App
PaaS(HCP)
IaaS
App App App
PublicCloud
App App App
PublicCloud
17confidential
Robustness HA & DR with e2e Security
1234
18confidential
Big Data River Security
o Authentication, Authorization‐ Kerberos, LDap, Radius, SSL‐ ACLs (Topics & IP), Role based access
o Encryption‐ Transport level SSL
o Five 9’s High Availability
o Flexible Disaster Recovery for async/sync replication
19confidential
Scale data bursts with“shock absorption”
1234
20confidential
Shock Absorbing the RIVER from the LAKE
Capacity &Availability Limits
Big Data RIVERShock Absorber
NetworkProcessingStorage
OutagesUpgrades
InconsistentAggregate
InputStream
21confidential
Big Data RIVER..deeper look
Big Data
PrivateCloud
PublicCloud
App 2App 1 App 3 App 4 App 6App 5 App 7 App 8
FrontOffice
RiskManagement
TradingEngines
Enterprise Data River
Compliance& Settlement
Easily extend your enterprise bus to the cloud
And capture enterprise events
22confidential
Big Data Case Study: Citibank Post Trade Buso “Rio” – the post
trade data river• Global, multi geo
post trade bus
• Lambda Architecture• Feeds Hadoop for analytics,
reporting and compliance• Feeds KDB for real time insights• Traditionally feeds into Netezza:
“Ocean” as data warehouse
o Solace is the Big Data RIVER “Post Trade Bus”
o 600M orders, trades, RFQs etc hit Solace every day,
o Solace feeds Hadoop, Netezza and KDB at different speeds for Analytics, AML, Risk, Compliance
23confidential
Other Big Data Rivers
24CONFIDENTIAL
Oh…one last thing….
25confidential
Big Data River
Enterprise Big Data River
AMQP
JMS
MQTT
REST
JMSAMQP
MQTT
SAP now supports nativeingest over MQTT…
