how to scale up applications - gridgain.com · how to scale up applications if covid-19 is causing...
TRANSCRIPT
![Page 1: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/1.jpg)
How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business
Nikita IvanovFounder & CTO, GridGain SystemsPMC Member, Apache Ignite
![Page 2: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/2.jpg)
2020 © GridGain Systems GridGain Company Confidential
COVID-19: Unique IT/Systems Challenges
• Up to 10x increase in demand or system load in a span of few weeks• Latency and throughput are both critical for minimal SLAs
Source: https://blog.griddynamics.com/rapid-response-to-covid-19-supply-chain-and-market-shocks/
![Page 3: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/3.jpg)
2020 © GridGain Systems GridGain Company Confidential
• Apache Ignite:– In-Memory Computing Platform– Distributed, Transactional, Write-Through Cache for RDBMS
• Concentrate today on:– Immediate effect– Easy-to-implement– Fit the long-term strategy
• Long-term solutions remain the same:– Heavy cost of re-architecting– Strong long term benefits
• Avoid short-term throw-away bandaids
Short-Term Solutions
![Page 4: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/4.jpg)
2020 © GridGain Systems GridGain Company Confidential
IMC Latency: Memory is Much Faster Than Disk
System Event Actual Latency Human ScaledOne CPU cycle 0.4 ns 1 sLevel 1 cache access 0.9 ns 2 sLevel 2 cache access 2.8 ns 7 sLevel 3 cache access 28 ns 1 minMain memory access (DDR DIMM) ~100 ns 4 minIntel Optane DC persistent memory access ~350 ns 15 minIntel Optane DC SSD I/O < 10 µs 7 hrsNVMe SSD I/O ~25 µs 17 hrsSSD I/O 50-150 µs 1.5 - 4 daysRotational disk I/O 1 – 10ms 1 – 9 monthsInternet: SF to NY 65 ms 5 years
![Page 5: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/5.jpg)
2020 © GridGain Systems GridGain Company Confidential
IMC Throughput: Massively Scalable
RDBMS
![Page 6: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/6.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Top 5 Apache Projects
Over 2M downloads per year and 4M total downloadsTop 5 Dev Mailing Lists
1.
2.
3.
4.
5.
Top 5 User Mailing Lists
1.
2.
3.
4.
5.
Monthly Ignite/GridGain Downloads
From January 1, 2019 Apache Software Foundation Blog Post: “Apache in 2018 – By The Digits”
A Top 5 Apache Software Foundation Project
![Page 7: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/7.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: In-Memory Computing Platform
Mainframe NoSQL HadoopDistributed Ignite Persistence
Disk Tier
RDBMS
Machine and Deep Learning
EventsStreamingMessagingTransaction
sSQLKey-Value
Service GridCompute Grid
Application Layer
Web SaaS SocialMobile IoT
Rol
ling
Upg
rade
s
Sec
urity
& A
uditi
ng
Mon
itorin
g &
Man
agem
ent
Seg
men
tatio
n P
rote
ctio
n
Dat
a C
ente
r Rep
licat
ion
Net
wor
k B
acku
ps
Full,
Incr
emen
tal,
Con
tinuo
us B
acku
ps
Poi
nt-in
-Tim
e R
ecov
ery
Het
erog
eneo
us R
ecov
ery
Distributed In-Memory Tier
GridGain Enterprise FeaturesApache Ignite Features
![Page 8: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/8.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Distributed Cache or Database
![Page 9: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/9.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Distributed Write-Through Cache
• Horizontally Scalable In-Memory Storage– ANSI SQL, ACID Transactions, MapReduce
• Automatic Synchronization With RDBMS– Write-Through - comparable writes performance– Write-Behind - faster writers, eventual
consistency
• 10-100x performance increase for reads
RDBMS
![Page 10: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/10.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Default Data Distribution
Canada
Toronto
Calgary
Paris
France
MarseilleMontreal
Ottawa
Country Table
City Table
![Page 11: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/11.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: SQL JOIN With Data Shuffling
Thick Client
Canada
Toronto
CalgaryParis
France
MarseilleOttawa
Montreal
ParisOttawaMontreal
1 & 4
2
2
3
1. Initiating Execution2. Execution on Servers (map phase)3. Data Shuffling4. Reduce Phase
![Page 12: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/12.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Co-Located Distribution (Affinity Co-Location)
Canada
Toronto
Calgary
France
Marseille
Country Table
City Table
MontrealOttawa Paris
![Page 13: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/13.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Configure Affinity Key
![Page 14: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/14.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Affinity Key to Node Mapping
Affinity Key Partition
Application Process Network Call
Node
CityRecord
![Page 15: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/15.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: High-Performance SQL JOIN
Thick Client
Canada
Toronto
Calgary
France
Marseille
1 & 3
2
2
1. Initiating Execution2. Execution on Servers (map phase)3. Reduce Phase
Ottawa
Paris
![Page 16: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/16.jpg)
2020 © GridGain Systems GridGain Company Confidential
Apache Ignite: Strong Consistency & Distributed Tx
• Strong or relaxed consistency– Transactional and atomic caches/tables
• Distributed ACID Transactions– 2-phase commit protocol
• Transactions supported for key-value APIs– MVCC for SQL transactions is in
experimental modeRDBMS
![Page 17: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/17.jpg)
2020 © GridGain Systems GridGain Company Confidential
• Open source• Data encryption in flight – SSL/TLS• Data encryption at rest
• Pluggable auth & auth• JAAS, LDAP, Active Directory, Kerberos, etc.• Cluster node authentication• Fine-grained authorization
• Multi-tenancy• Comprehensive auditing
Client Nodes
GridGain Cluster
In-Memory Data
Persistent Store
In-Memory Data
Persistent Store
In-Memory Data
Persistent Store
In-Memory Data
Persistent Store
GridGain: Enterprise Security
![Page 18: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/18.jpg)
2020 © GridGain Systems GridGain Company Confidential
Active-Passive Replication for
Disaster Recovery Split-Brain
Resolution
Active-Active Bi-Directional Replication
DC1 DC2
DC3
DC5
DC4
GridGain: Multi Datacenter Replication
![Page 19: How to Scale Up Applications - gridgain.com · How to Scale Up Applications If COVID-19 is Causing a Dramatic Spike for Your Business Nikita Ivanov Founder & CTO, GridGain Systems](https://reader033.vdocuments.mx/reader033/viewer/2022060309/5f0a705f7e708231d42ba346/html5/thumbnails/19.jpg)
2020 © GridGain Systems GridGain Company Confidential
Thank you & stay healthy!
ignite.apache.orgwww.gridgain.com