redis for horizontally scaled data processing at jfrog bintray
TRANSCRIPT
![Page 1: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/1.jpg)
XXX
HorizontallyscaleddataprocessingarchitectureUsingRedis
@
![Page 2: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/2.jpg)
About me
Ophir Hordan Member of Jfrog swamp Leading Bintray development
![Page 3: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/3.jpg)
Bintray – Leading Distribution platform – Support distribution of Maven,
Debian, RPM, Docker, nuget…. – Use CDN for fast download – Runs in multiple site around the
world
![Page 4: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/4.jpg)
Some Numbers ___________
Packages 300k
Downloads 900m/Month
![Page 5: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/5.jpg)
![Page 6: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/6.jpg)
![Page 7: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/7.jpg)
![Page 8: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/8.jpg)
Services
– Download binaries – Web application – REST API
– Backend micro services
![Page 9: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/9.jpg)
![Page 10: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/10.jpg)
The Layout
UI & REST API
MongoDB
Download Server
CouchDB
Index Calculation
Async Process
Event Processing
Event Processing
![Page 11: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/11.jpg)
No single point of failure
![Page 12: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/12.jpg)
True distribution
![Page 13: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/13.jpg)
Quiet period
![Page 14: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/14.jpg)
Data Sharding
![Page 15: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/15.jpg)
![Page 16: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/16.jpg)
and Twemproxy aka nutcracker Twemproxy
![Page 17: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/17.jpg)
The flow Service
Twemproxy
Micro Service
Twemproxy
![Page 18: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/18.jpg)
DB0InboundData
DB1Queuemanagementandquiteperiod
DB2WorkingDBandretriescounter
-ZsetbyEmestemp-Hashfordatapayload
HashkeysValuewithdatapayload
-Hashfordatapayload-HashforretrycounEngandrecoverydata
Redis structure LUAScanallkeysMovetodb1
zrangeByScore(currentEmestampminus5seconds)
![Page 19: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/19.jpg)
Service Pod Microservice Redis
Outbound Twemproxy
Redis
Pod
Pod
Inbound Twemproxy
Microservice
![Page 20: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/20.jpg)
Things we learned on the way… • Multiple values in single key is problematic to
monitor • aof file disk flush settings has big impact on
Redis performance • No official package of Redis and Twemproxy • Twemproxy only support single Redis db • Redis can easily hold Millions of keys • Protocol buffer works well as Redis document • Twemproxy doesn't have fail over • Twemproxy has limited sharding support (was
good enough for us)
![Page 21: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/21.jpg)
QUESTIONS?
![Page 22: Redis for horizontally scaled data processing at jFrog bintray](https://reader031.vdocuments.mx/reader031/viewer/2022030315/587c059c1a28ab03768b45a7/html5/thumbnails/22.jpg)