open banking with ocp msa and kafka · container images for apache kafka and apache zookeeper...
TRANSCRIPT
![Page 1: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/1.jpg)
Open Banking with OCP MSA and Kafka
Poste Case Study
OpenShift Commons - Milano 2019
Paolo Patierno, Principal Software Engineer - Red Hat
Pierluigi Sforza, Senior IT PM & Solution Architect - Poste Italiane
Paolo Gigante, Senior IT PM & Solution Architect - Poste Italiane
![Page 2: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/2.jpg)
[P. Patierno, Principal Software Engineer] @ppatierno
● Apache Kafka & Strimzi Intro
[P. Gigante - P. Sforza, Senior IT PM & Solution Architects]
● No more baby steps: All in! with the Open Banking ○ A common story○ A first bite! ○ Things become important○ Apache Kafka and OpenShift Architecture○ Strimzi test and considerations○ Our path, your takeaways
Agenda
![Page 3: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/3.jpg)
A Kafka cluster requires …
… a stable broker identity and network address… a way for brokers to discover each other
… durable state and storage recovery… brokers accessible from clients, directly
… and we also have Zookeeper
OpenShift provides …
… Statefulsets for stable identity and network… together with Headless services for discovery
… Services for accessing the cluster… Secrets and ConfigMaps for configurations
… PersistentVolume and PersistentVolumeClaim for durable storage
Apache Kafka on OpenShift : challenges
![Page 4: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/4.jpg)
Focuses on running Apache Kafka on Kubernetes ● Container images for Apache Kafka and
Apache Zookeeper● Operators for managing and configuring
Kafka clusters, topics or users
Provides Kubernetes-native experience ● Kafka cluster, topic and user as Kubernetes
custom resources
Open source project licensed under Apache License 2.0● CNCF sandbox project● THE way to deploy Apache Kafka for
cloud-native applications
You need an operator! Strimzi!
![Page 5: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/5.jpg)
Strimzi: Kubernetes Custom Resources
![Page 6: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/6.jpg)
Topic & Useroperators
Zookeepercluster
Kafkacluster
Cluster operator
KafkaCustom
Resource
How to deploy
![Page 7: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/7.jpg)
Topic & Useroperators
Zookeepercluster
Kafkacluster
Cluster operator
KafkaCustom
Resource
How to update
![Page 8: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/8.jpg)
Features
Scale Down
AffinityTolerations
Encryption
Authorization
Scale Up
Logging
Metrics
Healthchecks
Zookeeper
Off cluster access
Configuration
Source2ImageTopic
JVM Configuration
Authentication
Storage
HA
Mirroring
Kafka Connect
CPU and RAMUsers
Pod Disruption Budgets
Annotations
ImagePullSecrets
Labels
Upgrades
Secrets
ACLs Network Policies
Prometheus
Grafana
HTTP bridge
![Page 9: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/9.jpg)
https://www.poste.it/gamma/lettere.htmlMessaggio pubblicitario con finalità promozionale
![Page 10: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/10.jpg)
https://www.poste.it/gamma/spedizione-pacchi.htmlMessaggio pubblicitario con finalità promozionale
![Page 11: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/11.jpg)
https://www.poste.it/gamma/conti-correnti-bancoposta.htmlMessaggio pubblicitario con finalità promozionale
![Page 12: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/12.jpg)
https://posteassicura.poste.it/salute/poste-salute.htmlMessaggio pubblicitario con finalità promozionale
![Page 13: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/13.jpg)
… just loans at the moment but it’s an evolving market ;)
https://posteassicura.poste.it/cani-e-gatti/poste-amici-4-zampe.htmlMessaggio pubblicitario con finalità promozionale
![Page 15: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/15.jpg)
http://www.postemobile.itMessaggio pubblicitario con finalità promozionale
![Page 16: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/16.jpg)
src: http://www.poste.it (chi siamo, tab "I nostri numeri")
NUMBERS
Relazione Finanziaria Annuale al 31/12/2018
![Page 17: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/17.jpg)
A COMMON(S) STORY
Many products, many users! daily, Lines Of Business push IT for quality and delivery speed
![Page 18: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/18.jpg)
A FIRST BITE!
a stack upgrade plan (EAP4-5 to EAP7) for 7 monolithic apps was used to test and prototype a DevOps model, toolset and... OpenShift!
DevOps and OpenShift 3.9 for NSA
![Page 19: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/19.jpg)
JAVA S2I FOR OPENSHIFT
A SECOND BITE!
Aggregated view of entities using upsert datastream from many sources
OneView App … DevOps, OpenShift && Kafka were there!
Spring REST
Presentation controllers
Mongo Client
Ingestor controllers
Kaf
ka C
lient
Por
met
heus
Exp
orte
r
Custom Connector
Cange Data Capture
on-going
Will it works?! …. 500 mln of records ingested during first 8 hours GOT IT! :)
![Page 20: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/20.jpg)
THINGS BECOME IMPORTANT
The core financial app for the Open Banking now runs on OpenShift and Kafka… Poste got there on time and with the a better response time (in order of magnitude!)
src: linked-in
...
![Page 21: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/21.jpg)
THE THIRD BITE!
Mic
rose
rvic
es
API GATEWAY IDP
SCA
TPP
Legacy
Payments
AntifraudDevOps
Metrics
Tracing
CICS SISEB3ANAG CARD
Anagrafe Accounts Cards Payments
Movements Fraud Mg Registry Utils Foundscheck
PSD2 DETAIL
![Page 22: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/22.jpg)
PSD2 GEO REPLICA
ROME
cold stdby (async replica)
1 KmAZ1 (Europa) AZ2 (Congressi)
TURIN
active active services
WrapperSwitch
Arbiter
Mirror Maker
![Page 23: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/23.jpg)
AFTER < 1.0 Y
a fast changing company
13 clusters (OCP, OKD 3.9 and 3.11) with a production of 1300 core
15 initiatives in develop stage, 4 systems in production
3 pipelines per project, 1200 jobs
863 repos, 353 developer
3 datacenter 1 Cloud Provider
![Page 24: Open Banking with OCP MSA and Kafka · Container images for Apache Kafka and Apache Zookeeper Operators for managing and configuring Kafka clusters, topics or users Provides Kubernetes-native](https://reader033.vdocuments.mx/reader033/viewer/2022052800/5f10c5827e708231d44abd60/html5/thumbnails/24.jpg)
Kafka (on bare metal) vs Strimzi
+ Efficient Bare Metal deployment+ Massive Cross App communication+ Resilient traditional storage
- Expensive- Hard to maintain - Hard to scale
+ Fast containerized setup+ Easy inter-app async comm+ Quick scale-up/out
- ?! hopefully none! :)
Test running!Be tuned for next
Commons