real world enterprise reactive programming using vert.x
DESCRIPTION
The presentation is about real world and production ready example in reactive programming area, using Vert.x. It shows the best practices, event driven application architecture on the cloud and lessons learned.TRANSCRIPT
Mariam Hakobyan, Sascha Möllering, Björn Stahl
Real World Enterprise Reactive Programming using Vert.x
| zanox AG
TABLE OF CONTENTS
1.Introduction
2.The Beginning
3.How to start?
4.Best Practices
5.Architecture
6.Deployment
7.Vert.x module system
8.Integration with messaging system
INTRODUCTION ZANOX
Europe‘s leading performance advertising network
THE BEGINNING
Java Magazin 04.14: Vert.x im UnternehmenseinsatzEntwicklung und Betrieb von asynchronen Applikationen mit Vert.x in der Praxis
THE BEGINNING
● New core system for zanox
● Requirements are pretty high (not negotiable):
●Low latency
●High throughput
●Reactive
●Fast
THE BEGINNING
https://github.com/Mr-Steel/vertx_fatjar
“Vert.x is a lightweight, high performance application
platform for the JVM that's designed for modern
mobile, web, and enterprise applications.”
Vert.x
WHAT IS VERT.X?
WHAT IS VERT.X?
Polyglot
WHAT IS A VERTICLE?
●Classes with an own Classloader
●operates Single Threaded
●executed by an Event Loop
Event Loop Pool
Event Bus
V V V W W W
TAKE A LOOK INSIDE
HOW TO START?
●mvn archetype:generate -
Dfilter=io.vertx: (do not forget the
colon!)
●generates structure for all languages (JS,
Ruby, Groovy, Python)
●maven pom is already set with relevant data
HOW TO START?
Lessons learned - Profiling
● Verticles and WorkerVerticles appear in different
subtrees
● Be careful with blaming, a lot of stuff is just waiting
● If you think a verticle is a bottleneck: start more
instances of this verticle
● Measure every change & compare results (yeah yeah -
but really - do it!)
BEST PRACTICESStarter Verticle
BEST PRACTICESLogging
BEST PRACTICESLogger Verticle
●Disadvantages:
●log entries are not in order
●loss of log levels
●Solutions:
●use IDs to follow log message flow
●use one LoggerVerticle per log level
BEST PRACTICES - On the fly deployment
BEST PRACTICES - On the fly deployment
●Do not block the loop
●put blocking code or extensive computation into
worker verticles
●Keep the application responsive
●stress test as often as possible
●Encapsulate common code in modules (more
on this later)
BEST PRACTICESIn General
fat-jar: one package to rule the deployment
BEST PRACTICESDeployment
BEST PRACTICESDeployment
zip-File:
●have a separate folder for scripts
●bind packaging to verify phase to have it all
with one command “mvn verify”
●complete example is on GitHub
Internet
Auto Scaling group
Auto Scaling group
Availability Zone
Availability Zone
Architecture
Architecture
●Main framework Vert.x
●mod-kinesis (Kinesis-Adapter for Vert.x)
●AWS Java SDK
●IAM roles for Amazon EC2 instances
●Coda Hale metrics and CloudWatch reporter
●Jedis (Redis client for Java)
●…
Deployment
WorkstationGitHub Jenkins
Python/Fabric
Nexus
Datacenter
Artefakt
Deployment
Deployment
EC2 instance with Dockerregistry (port 5000)
S3 bucket to store Docker images
Deployment
Amazon Cloud
WorkstationGitHub Jenkins
Python/Boto
Docker Image
Docker RegistryS3 bucket withDocker Images
Deployment
Deployment
Deployment
Deployment
python start_docker_instance.py \
-r <myregistry>:5000 \
-i kinesis-producer \
-t 95 \
-q 2 \
-s quality \
-d '-d -p 8080:8080'
https://github.com/SaschaMoellering/aws-docker-scripts
Deployment
python start_elb.py \
-r <myregistry>:5000 \
-i kinesis-producer \
-t 95 \
-s quality \
-d '-d -p 8080:8080'
https://github.com/SaschaMoellering/aws-docker-scripts
Deployment
Deployment
Vert.x Module System
Vert.x Module System
Benefits:
● Re-usable components as a zip file
● Maven and Bintray integration
● Vert.x module registry
● Vert.x downloads and installs modules for you
Vert.x Module Registry
Vert.x Event Bus
● Address is a simple string
● Types - Pub/Sub & P2P
Pub/Sub
P2P
Sender
Receiver
Integration with Messaging System
KAFKA MESSAGING SYSTEM
●Apache Kafka is a distributed publish-subscribe messaging system
●Fast
●Scalable
●Durable
●Distributed
Integration with Messaging System
Main Terminology:
●Topic
●Producer
●Consumer
●Broker
Kafka Module in Module Registry
Open source Kafka module in Vert.x’ module registry● http://modulereg.vertx.io/
Using Vert.x Module
// programmatically
// command line
Using Kafka Module
Configuration is a JSON object:
StatsD Support
Config
Metrics of Vert.x Project
On 4 Cores virtual machine we had the following
results:
●~28 K requests per second without Kafka,
with lookup from Redis
●~18 K requests per second with Kafka and
lookup from Redis
Metrics of Vert.x Project
Test Traffic ~1000-3000 requests per minute:
●mean response time: 1-2ms
●90% of responses 0.86 ms
●fully cached: 0.65 ms
●CPU Load: 1-3%
●Memory: 90MB (300MB reserved)
GET INVOLVED
Kafka Module - https://github.com/zanox/mod-kafka
Kinesis Module - https://github.com/zanox/mod-kinesis
https://github.com/Mr-Steel/vertx_fatjar