osgi and financial messaging - a successful use case - luis matos

© 2015 D+H USA Corporation. All rights reserved. Osgi and Financial Messaging: A successful use case. Osgi Conference Nov 5th, 2015

OSGI And Financial Messaging:A Successful Use Case

Luis Festas Matos D+H BBP GmbH


Luis Festas Matos

Emails: [email protected] , [email protected], Linkedin: https://ch.linkedin.com/pub/luis-festas-matos/20/b75/66a

Brazilian, from Rio de Janeiro

Java Engineer since 2002

Osgi Engineer since 2010

Likes samba, hot weather, beachesLives in Switzerland

FACTS


Largest SWIFT ServiceBureau worldwide • Operation since 1995 • Serving more than 500 customers in over 30 countries

directly

Company

D+H

Nr.1

1984BBP Group • D+H BBP GmbH, Switzerland (1984) • D+H Financial Messaging, UK (since 2008) • D+H GmbH, Germany (since 2011)

Pan-European locations • Switzerland (Baden near Zurich and Geneva) • Great Britain (London and Nottingham) • Germany (Frankfurt) Cross-border resilience • Two data centers in Switzerland and Great Britain


Customers: Europe and World

500+ Financial Institutions and Corporates


More than 500 Clients in over 30 CountriesFinancial Institutions

Corporates

ACH (Automated Clearing Houses)

ASP / BSP


Integrator: Software layers

• Connectors: Java, C++ • MQ, File-Based, TCP/IP

SW

IFTG

MP

Back

end

Corp

orat

e ST

P Ban

k S

TP

Connector

OSGi Core

• Core: OSGi

• GMP Backend, Corporate STP, Bank STP Systems, SWIFT, FED/CHIPS, others

OthersFED/


Global Messaging Plus (GMP)

Customer’s STP

Processing Hosts

Interbank Financial Services

Hub (Osgi Core)

Archive Alarming

Monitoring

SWIFT Funds

Global Compliance Reconciliation

Multiple provider Data feeds

JBoss SWIFT

Application

JBoss SIC

Application

JBoss …

ApplicationSWIFT Files

SWIFT FIN

SIC

SECOM

FED Chips

Tomcat / Zkoss

Users

Performance • Over 400’000 transactions per hour / per OSGi

core Platforms • Red Hat, Linux

Web Manual Messaging • Users • Tomcat / Zkoss

• Interbank services (SWIFT, FED, CHIPS, SIC, ..)

Interfaces to … • Hosts

• Security & alarming management systems• Archiving- and reconciliation applications

Hub (OSGi core) • Queuing, routing • No loss of messages • No duplicates • Conversions • Logging, audit trail • Automatic recovery

• JBoss Backend Applications


Prehistoric Software • Limited customer-friendly API

• Not fast enough for more demanding customers (1 million msgs/hour max with 2 cores)

• C++ STP Core

• Limited architectural plan (mostly do as you go)

• Non-scalable (maximum 2 STP cores per Backend)

• Fat Java client

• Reliable but not transactional


GMP requirements

• Transactional / No message loss

• Easy maintenance / deployment

• Automated / customizable workflow

• Continuous integration tests

• Extendable by customers

100%

99.94%

100%2006: 99.92%

2007: 99.91% 2008: 99.94%

100%

100%

99.98%

2012: 99.98% 2013: 99.99% 2014: 99.98%

R 1 3 5

2 4 6

• Faster but keeping same level of stability

• Keep virtually 0% downtime

• Scalable vertically…and horizontally

• Make free coffee

• Guess Lotto numbers

© 2015 D+H USA Corporation. All rights reserved. 10

2010: Research

Lean Architecture

Vertical and Horizontal scalability

Fast

No message loss

Automated AND customizable workflows

Open source Support from vendors

Easy maintenance / deployment


Proofs of Concept & Benchmarks

• J2EE architecture in general

(too slow)

(too slow, no easy horizontal scaling)

( namespace, classloader problems )

( not XA transactions ready)

( too slow )

( not OSGi ready )

( too heavy and slow )

• Servicemix with JBI and NMR, Aries JPA

• Jboss with JPA / JTA

• Spring DM

• Cassandra, CouchDB, noSQL DBs

• Drools

• Hibernate

• Workshops with Fuse / Jeff Genender and Adrian Trenaman


Survivors Development Support Tools

Development Tools / Platforms

Database Systems

Operative Systems


Architectural / Design decisions • Transactions with XA (no JTA)

– Camel, ActiveMQ, Database and Hazelcast as XA resources

• Develop own «non-sql like» lean persistence layer (no JPA)

• Camel used for main routes connecting services – Routing slip pattern for flexibility

• Mvel rules on strategic places for customization • Hazelcast for inter-karaf distributed data

• Aries blueprint XML – Injection and service creation, life cycle (0% annotations)

• Separate Processing database from Querying database – 1 simple index per table (key) on processing database (speed optimized) – Several indexes on querying database (Solr based, outside Integrator)

• Karaf «features» for provisioning


Topology

CO

NN

ECTO

RSNetwork of Brokers

HOSTS

FINANCIAL SERVICES


Message Flow on OSGi

XA

XA

TRANSACTION WORKFLOW

Printer

Query

DB

Final Destination

Exceptions

XAXA

SERVICES

CAMEL ROUTESMVEL RULES


Key differentiators: JSON• Jsonschema2Pojo-based DTOs

• Very fast – on-the-fly Json to Pojo to Json conversion – using FasterXML’s jackson-databind – Github contributions to OSGify project

• Used in: – Intra-process communication (JSON over ActiveMQ) – Key-value, No-SQL-like persist

• Collections (tables) inferred directly from POJO definition – Own created annotators – Using advanced JSON (Jackson) serialization features


Key differentiators: Interface• Own defined persistence layer

• Bare, Map-like interface

• “Need for speed”


Key differentiators: Transactions• XA transactional achieved

• Cutting edge technology – Often using latest or even SNAPSHOT releases – Pushing vendors to be full XA-ready – Close collaboration with Atomikos and Hazelcast

• XA resources – Camel – Oracle, H2, Ingres DBs – ActiveMQ – Hazelcast

• Constant tests – “Pull the plug” – Recovery while system online – Recovery under system restart


Key differentiators: Flexibility• Versioned Mvel Rules

• Changeable by the customer – New rules can be added without downtime – Switch between old and new rules by command

• Friendly GUI for BPMN-like processes to compile to Mvel rules – In development


Conclusions• Open source-based, support contracts • No message loss (tests and customers)

• Transactional (with XA)• Fast (suprisingly faster than C++-based core, one-to-one)

• Stable (downtime < 0.01%)

• Customizable with Mvel Rules

• Out of the box, automated basic configuration

• Lean architecture

• Vertically and horizontally scalable (karaf and ActiveMQ)

• Brewing coffee or guessing the Lotto numbers


21

THANK YOU

Luis Festas Matos D+H BBP GmbH

osgi and financial messaging - a successful use case - luis matos

Technology