fmdphome.fmdpglobal.com/wp-content/uploads/2016/02/... · fmdp excel integration (emil) and api. 2....
TRANSCRIPT
BCC Group International
White Paper
Issue 5.2 November 2015
[FMDP]GLOBAL Financial Messaging Data Platform
Open Access to Global Information
Entitlement | Flexibility | Performance | Reliability | Scalability | Resilience | Cost Reduction
Copyright © BCC Group International GmbH & Co. KG 2008 - 2016
This BCC Group document is for BCC Group customers and authorized developers, and is provided for such
information purposes only.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
2
TABLE OF CONTENTS
1. Preface ............................................................................................................................................ 3
2. Management Summary .................................................................................................................. 3
3. A choice between two unsatisfactory options ............................................................................ 6
4. Overview of FMDP Approach ........................................................................................................ 7
5. FMDP Design Goals ....................................................................................................................... 8
5.1. Dedicated Hardware for Messaging ......................................................................... 8
5.2. Point-to-Point TCP/IP Protocol ................................................................................. 8
5.3. Topic-based Data Delivery ........................................................................................ 9
5.4. Hardware-based Entitlement Firewall (MECS) ......................................................... 9
5.5. Single API ..................................................................................................................10
6. FMDP Components ...................................................................................................................... 11
6.1. Feeds & Caching ......................................................................................................11
6.1.1. Broadcast Feeds .....................................................................................................11
6.1.2. On-Demand Feeds ..................................................................................................11
6.1.3. Contribution / User Publishing ...............................................................................11
6.1.4. Cache .......................................................................................................................11
6.1.5. Message Recorder ..................................................................................................11
6.2. Multi-vendor Entitlement Control Service (MECS) .................................................12
6.3. Vendor-approved Entitlement Control ....................................................................13
6.4. API .............................................................................................................................13
6.5. Testing Tools ............................................................................................................13
7. TREP-RT/EDGE/RMDS Integration ............................................................................................. 14
8. Bloomberg BPipe Feed Handler ................................................................................................. 14
9. Microsoft Excel® Integration (EMIL) .......................................................................................... 14
10. openMAMA Bridge ....................................................................................................................... 14
11. ITRS GENEOS Monitoring ........................................................................................................... 14
12. Cloud capabilities ........................................................................................................................ 14
13. Example FMDP Setup (not showing DR site) ............................................................................ 15
14. BCC Group – FinTech Made in Germany .................................................................................. 15
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
3
1. PREFACE
This Whitepaper describes the FMDP System and why the financial market in
particular requires a new approach to distributing messages (e.g. market data). It
provides an overview of the current market situation and describes the approach that
BCC Group has chosen to implement a new generation Message Distribution
System, an approach which keeps all the useful features of existing architectures, but
enables greater speed and simplicity.
As a source-agnostic platform, FMDP focuses on cost reduction, transparency and
governance while lowering the TCO. It provides an easy, low-risk migration away
from legacy platforms while offering royalty-free connectivity of feeds and
applications.
FMDP architecture was designed and developed by BCC Group with important
development contributions in the Entitlement arena by Royal Bank of Canada
Capital Markets (RBCCM) supported by Solace Systems. Testing and production
stability contributions (besides RBCCM production tests) were made at the premises
of German DZ BANK during a four-year period (2011-2015) of active production
support by DZ BANK market data engineers and developers who also heavily tested
FMDP Excel Integration (EMIL) and API.
2. MANAGEMENT SUMMARY
Users, specifically traders all over the world, can communicate in real time about their
problems. Most of their systems have not seen any enhancement in flexibility or
speed for over 10 years. They also have ongoing, long discussions with IT when it
comes to communication enhancements e.g. additional symbols, additional fields,
new recipients or sending values out of spreadsheets.
Spreadsheets are a good example of how difficult it is to get all vendor and customer-
owned data into a single application or, worse still, publish proprietary values. It is not
enough that each vendor requires their own Add-In, but some of the Add-Ins even
seem to intentionally conflict with each other. And the data remains in encapsulated
worlds of the transporting vendor although it is non-vendor data.
Market data vendors have been highly successful in binding their customers to their
systems by implementing terminals connected to their communication channels
(including Excel). After all the crises the financial world has undergone, users are
either connected to one communication channel or the other; in rare cases you are in
fact connected to more than one.
Basically the only “professional” communication channel that everybody can use from
their workplace still remains email. It is available to everybody but is not specifically
designed for communicating values, value groups or news in a structured, context-
sensitive way and it is neither smart nor real-time.
While the communication technology world has evolved to offer amazing
communication freedom using mobile devices and the internet, the trader world is
basically the same. Ultra-high speed ultra-high available messaging technology
burned into chips of appliances (e.g. Solace Systems message routers) have in many
cases, taken the place of overpriced and old communication architectures that require
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
4
operating system licenses for hundreds of servers together with messaging software
implying exorbitant license and maintenance costs.
Not so in the dealing rooms where global financial data distribution has not yet
reached that level of freedom.
Despite the fact that enterprise data feeds from external vendors (if not feeding into
their own platforms) also provide global distribution for proprietary customer data at
very high cost, they are mostly used to feed directly into applications.
Traders are restricted in content and update rate when publishing over a provider
channel and they are still trapped inside their “preferred” provider network together
with the provider’s terminal.
None of them is able to feed into a vendor agnostic platform that would be capable of
distributing everyone’s data altogether in a unified and entitled way to all applications,
including Excel.
FMDP has been developed for these very reasons, specifically to provide trading
room communication freedom.
FMDP development started 2009 with an early production version in 2011 and the
fully functional version was tested with Bloomberg and Thomson Reuters RMDS
Bridged Data in 2012. Today FMDP is a fully tested and stable, ready to use and fast
in implementing Market Data Distribution Platform for all kinds of data. FMDP
simultaneously supports thousands of users and millions of subscriptions.
Combining multiple FMDP Installations in different locations that are interconnected
allows the distribution of data globally without overloading existing communication
lines, because only subscribed data portions from the available set of data from a
distant location will be transferred.
FMDP was intended to meet the following requirements and these have been 100%
achieved:
Open, bank-owned and giving free but entitled access to all market data
providers and applications,
Designed to be able to use vendor-specific entitlement systems if requested
by the vendor or exchange, but also having an open entitlement system for all
data sources and applications that cooperates seamlessly with any vendor
entitlement system,
Designed to distribute high volumes of messages with low latency,
Easily scalable to higher capacity by just adding a piece of hardware,
Distribute and entitle real-time and reference data,
Include all components for 24/7 operation and resilience,
Designed to be “cloudable” and globally deployable.
Small footprint and low maintenance cost
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
5
Designed as a product, not as a project, in order to achieve street-wide
acceptance and market data community support, incl. market data vendors
FMDP meets these requirements as follows:
FMDP is hardware based (FMDP routers are Solace Systems message
routers). This optimized and high-performance hardware enhances speed by
several orders of magnitude (factor 10-1000) compared with software-based
messaging solutions and reduces the number of servers. Reducing the
number of servers means reducing system complexity and saving software
licenses and production costs.
FMDP enables vendor independence and thus more flexibility by providing a
universal interface (FMDP API) for all feeds and applications.
FMDP allows entitled distribution of any kind of data object, obviously real-
time market data messages, but also any type of serialized object type, thus
providing entitled distribution for real-time data, historical and reference data
in all formats.
As an intrinsic part of FMDP, resilient failover incl. last value caches is
provided as a stand-alone process or built into feed handlers.
FMDP provides entitlement clients for managing and monitoring the system as
well as user-to-data and user-to-application entitlement management.
A message recording service is available for all data sources using modern
NoSQL databases.
FMDP is “Cloud Ready”, which means that all components can also be run in
a private or public cloud if required.
FMDP installations in different locations can be combined into a global,
optimized distribution network.
The universality of the FMDP API offers significant simplifications in
developing market data applications for real-time, reference and historical
data.
An Excel Add-in is available for sophisticated subscription and publishing over
the platform globally.
FMDP component architecture is transparent and easy to install
FMDP architecture was designed and developed by BCC Group with
important development contributions in the entitlement arena by Royal Bank of
Canada Capital Markets (RBCCM) supported by Solace Systems.
FMDP’s Hardware-based, High-Performance Message Distribution Service was
developed to overcome the trend of ever-increasing infrastructure demands by
focusing on introducing new hardware technologies that include street-wide
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
6
accepted entitlement and the provision of a smooth migration path to the new
platform.
BCC Group today owns and maintains all parts of FMDP including contributions
from RBCCM.
Image 1: Current World of Distribution with Broadcast Backbone in the financial sector
3. A CHOICE BETWEEN TWO UNSATISFACTORY OPTIONS
Message distribution is a massive I/O problem that cannot be solved with faster
processors or more memory. The bottleneck is in getting data to the applications,
while supporting different service levels. In today’s installations, hundreds of servers
with hundreds of processors manage the distribution. This requires considerable
resources such as electricity, rack space and manpower to administer the
infrastructure, together with the enormous expense of the operating system and
messaging software licenses, their updates, customization and maintenance.
Altogether, in-house tailored or standard front, middle and back-office applications
can account for up to 80% of all messages, mostly internal data. Consequently there
are two options:
Buy everything from one supplier to avoid the complexity and cost of
connecting applications to different providers. Homogenous infrastructure
avoids multiple bills and keeps IT costs to a minimum. Unfortunately, banks
are then restricted to that supplier’s universe of data, infrastructure and
applications, and it is unlikely that they will cover all their information needs or
be first in adapting to new trends. Once trapped in the provider’s universe,
banks are subjected to arbitrary mark-ups dictated by the provider.
Connect applications directly to different providers, offering the freedom
to choose the best available information sources, and making it possible to
negotiate costs. However, this results in non-transparent billing, additional
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
7
exchange fees and high IT maintenance costs. Overall reliability and
manageability may suffer leading to unpredictable costs for support and for
further development of home-grown solutions.
Image 2: Current World of Data Delivery with direct Feed Connections to Applications
So today, companies in the financial industry have to choose. On the one hand, opt
for a complex and highly expensive single-provider solution that does not satisfy the
needs of all groups in the bank. Alternatively choose a home-grown consolidation of
feeds from different providers that forces them to build application connectors and
operate and maintain highly complex IT infrastructure and hundreds of program
sources. Even worse, it may be a mixture of both approaches.
4. OVERVIEW OF FMDP APPROACH
Consolidation of the provider scene has effectively reduced the number of market-
data platforms available. The TCO of a new solution must therefore be significantly
lower than the current installation, and must be vendor independent, i.e. it cannot be
developed or sold by a financial market data provider. There must also be a way to
migrate applications incrementally, without requiring a “big-bang” approach. So, any
new solution must be capable of coexisting with the pre-existing systems until all
applications have been moved.
FMDP handles the main task of ultra-fast I/O with innovative purpose-built hardware
instead of software-based solutions running on general-purpose servers like current
products. This addresses most of the problems mentioned above by eliminating
technical dependence on data vendors, removing the necessity for server proliferation
to meet scaling requirements, providing a single platform for all service levels, and
reducing administration tasks required to maintain the infrastructure.
As one of the most important features of a market data distribution platform, FMDP
provides entitlement as a key feature, giving customers the ability to control which
clients are permitted access to which data. An important design goal was to build a
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
8
system where all sources can be connected and all providers and exchanges will be
happy to allow the distribution of their data (Entitlement).
Where vendors require the use of their own Entitlement systems, the feed handlers
perform checks on them before feeding the data into FMDP.
Due to its simple architecture enabled by the use of Solace Systems appliances,
FMDP is a fast-to-install solution that includes an Entitlement controlled Pub/Sub API
along with a ready to go Excel Add-In.
5. FMDP DESIGN GOALS
5.1. Dedicated Hardware for Messaging
To address all the requirements e.g. for a new generation market data distribution
platform, FMDP is built around dedicated hardware for high speed data distribution
and filtering (FMDP-Blade) running on devices from Solace Systems
(www.solacesystems.com). This hardware is purpose-built to offer low latency, high
resilience and high throughput of streaming, topic-based data without the need for
servers, operating systems or additional messaging software.
FMDP has added special built-in features for topic-based access control – the
Entitlement Firewall.
Today, the most time and performance critical components of a data distribution
system are covered by a number of communicating servers all connected to one or
more broadcast backbones. Solace hardware has “burned” this publish/subscribe
paradigm into pure hardware using FPGA technology that, in contrast to standard
CPU processing, can speed up routing and delivery of data by 10 to 1000 times.
Using Solace hardware, all data is directly routed to the subscribers and the
broadcast layer is no longer needed, thus making latencies predictable having
removed the need for a dedicated broadcast backbone. Furthermore this approach
perfectly supports forced entitlement capabilities.
A single FMDP-Blade is capable of managing up to 20 million messages per second,
and can deliver wire-to-wire latency as low as 75 μsec. The hardware approach
makes it possible for FMDP to handle millions of subscriptions to thousands of clients
simultaneously.
5.2. Point-to-Point TCP/IP Protocol
All communication in FMDP is based on TCP/IP point-to-point services with domains
separating vendor streams from each other and different accounts with hundreds of
consumers.
The FMDP Domain capabilities of the system allow for multi-client installations in
remote data centers for hosting or cloud constellations.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
9
Image 3: Old World global Distribution vs. FMDP based on Solace Hardware
5.3. Topic-based Data Delivery
All data received is transformed to a standard Solace in-memory format and placed in
the FMDP-Solace appliance for fast DMA access of subscription management and
topic routing on the FMDP-Blade.
All administrative communication, i.e. subscription requests, authentication requests
or feed connection requests are TCP/IP messages controlled by the FMDP API. This
ensures data is accessed only by entitled and authenticated participants. Data is
tagged with an internal “Product Key” derived from the provider’s unique information
about the data’s provenance, asset class etc. to ensure Entitlement capabilities.
In addition, all FMDP admin services are directly connected to the FMDP hardware,
avoiding the need for different technologies, protocols or proprietary software
components for interconnection, so all admin features also benefit from Solace’s built-
in capabilities for resilient, distributed infrastructures.
5.4. Hardware-based Entitlement Firewall (MECS)
MECS (Multi-source Entitlement Control Service) adds all components to the client’s
FMDP Domain that manage the Entitlement Firewall for consumers; i.e. for
authentication, entitlement and connectivity of feeds and applications.
Subscription attempts by the API are checked and if there is a positive response the
data is passed to the consumer.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
10
Image 4: Solace Hardware based Entitlement Firewall
NOTE: It is important to note that entitlement is also checked on initial
subscription request to some vendors in the feed adapter if the vendor so
requires. This is the case in the Bloomberg and Thomson Reuters feed
handlers. The Bloomberg Feed Handler verifies user subscriptions against the
EMRS while the Thomson Reuters Feed Handler uses openDACS for user
entitlement IN ADDITION to the FMDP Entitlement. This is due to the fact that
FMDP Entitlement cannot be turned off by design.
5.5. Single API
FMDP delivers a royalty-free API that allows all types of participants to connect to the
system either feeding or consuming. MECS ensures that all data is delivered to
entitled users only while freeing the developer from complex handling of entitlement
functions. There is only one API for all communication within FMDP, i.e. Admin Tools,
Management Clients and Monitoring all use the same communication paradigm.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
11
6. FMDP COMPONENTS
FMDP calls all kind of connected applications “Participants”. These are distinguished
by their role in the system and are differently supported in the FMDP API. Entitlement
is also handled differently depending on the role defined for the application. An
application has to be previously defined, incl. its role, to be able to login.
6.1. Feeds & Caching
The Feed participant is an application that publishes into the FMDP System. There
are different types of feeds as described below. All feeds can contain a last value
cache depending on usage.
6.1.1. Broadcast Feeds
Broadcast Feeds are feeds that start pumping all available data to the FMDP System
on connection to the delivering provider. Examples are the former Bloomberg BPipe,
VWD Broadcast feed and Morningstar Real-Time Broadcast feed (all TPC). FMDP
supports the connection of Broadcast Feeds and manages the cache automatically.
6.1.2. On-Demand Feeds
On-Demand Feeds deliver their data on request. As such, only data that has been
previously subscribed to is delivered. Examples are Bloomberg’s new B-Pipe and
Morningstar Real-Time on-demand feeds (TAPI), Thomson Reuters EDGE devices
as well as the RMDS or TREP-RT System interconnection (H2RBridge). FMDP allows
the connection of on-demand feeds, handling the initial subscription to the provider’s
feed on first request by a consumer, and managing the cache automatically.
Depending on data vendor restrictions on data access, the vendors’ Entitlement
Systems are used for Entitlement (e.g. openDACS or EMRS).
6.1.3. Contribution / User Publishing
FMDP treats contributors as feeds. All rules for feeds apply to contributions in the
same way, i.e. contributions must deliver Entitlement IDs and must belong to a
registered feed. MECS manages permissions to specific sources and message types.
6.1.4. Cache
FMDP includes high speed dynamic 64-bit caches for dynamic last value
management. FMDP high-speed caching technology is used for stand-alone caches
serving e.g. Contribution Data from EMIL (FMDP Excel Integration) or is included in
feed handlers. Caches include data persistence, automatic failover, lookups, cron
controlled clean-up and many more features. Caches are handled similarly to feed in
FMDP thus their role must be “Feed” type.
6.1.5. Message Recorder
FMDP optionally includes 64-bit message recorders for any kind of message type into
a NoSQL Database. This can be required for compliance reasons or for building
histories for internal data. The message recorders are normal applications in FMDP.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
12
6.2. Multi-vendor Entitlement Control Service (MECS)
The Entitlement Firewall is managed by MECS. MECS is described in more detail in a
separate document. FMDP has an entitlement system in addition to the authentication
mechanism, namely the Multi-vendor Entitlement and Control System (MECS).
Both are integrated in the FMDP MECS service and rely on the FMDP database. Its
most important tables are:
Users: lists all users and applications permitted to connect to the FMDP
environment.
Products: lists all data and application products available in the FMDP
environment, e.g. feed servers, feed record types or programs (consumers or
applications).
UserEntitlement: contains the entitlement information for users and products.
DataDictionary: contains field descriptors for fields of a specific provider and
feed ID (optional).
Multiple FMDP MECS services may run concurrently for fail over and load balancing
purposes, but all must use the same database. MECS is delivered with two graphical
client applications for intuitive management of the FMDP Platform:
FMDP Manager
GUI for the management of all components and settings for FMDP
Image 5: FMDP Manager
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
13
FMDP Entitlement Editor
GUI for the intuitive management of all Data and Application Entitlements.
Image 6: Entitlement Editor
6.3. Vendor-Approved Entitlement Control
For sources that require their own entitlement system (e.g. Bloomberg EMRS) a
vendor approved entitlement agent is available.
6.4. API
FMDP is delivered with a royalty-free API, the same API that is used for all existing
FMDP Components. The API allows subscribing, publishing and request/response
programming for application developers while assuring Entitlement checks and
automatic monitoring without user intervention.
6.5. Testing Tools
A sophisticated set of installation testing and development support tools such as
message dumps and mock publishers are available for FMDP.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
14
7. TREP-RT/EDGE/RMDS INTEGRATION
A high speed FMDP-to-TREP-RT-Bridge (H2RBridge) is available that allows all
applications connected to FMDP to receive TREP-RT Data from all active TREP-RT
sources in high-speed. Entitlement is then managed by openDACS.
8. BLOOMBERG BPIPE FEED HANDLER
For FMDP (and raw Solace Systems devices) a Bloomberg Feed Handler is available
that allows all applications connected to FMDP to receive Bloomberg Data. The
Bloomberg BPipe Feed Adapter has been certified by Bloomberg for RBCCM
worldwide locations and uses EMRS entitlement.
9. MICROSOFT EXCEL® INTEGRATION (EMIL)
A fully featured FMDP-Connector for Subscribing and Publishing in high-speed to
FMDP from and to Excel® (Excel® Multi-source Integration Layer, EMIL) is available.
10. OPENMAMA BRIDGE
FMDP is fully openMAMA connected. All openMAMA applications can immediately
use FMDP including all features such as caching and entitlement.
11. ITRS GENEOS MONITORING
ITRS GENEOS Monitoring is available for all FMDP Feed Handlers.
12. CLOUD CAPABILITIES
FMDP components can be deployed to private clouds thus making the Platform
available to different firms simultaneously if needed. However, feed adapters and
entitlement components must be installed and entitled on a client-by-client basis, but
all can use the same Solace HW. For each client the components run on different
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
15
FMDP-Domains. FMDP-Domains can still be configured to interconnect different
clients for exchanging client’s internal data in high speed.
13. EXAMPLE FMDP SETUP (NOT SHOWING DR SITE)
14. BCC GROUP – FINTECH MADE IN GERMANY
BCC Group is a Frankfurt-based FinTech software house specialized in the
development and delivery of high-quality software solutions for the financial industry.
For nearly two decades, we have been supporting customers in the area of Market
Data Technology, Market Data Processes and Spreadsheet Integration Technology.
BCC Group’s core competence is the development of solutions based on financial
market data feeds (Bloomberg, Thomson Reuters, Morningstar, etc.) and Solace
Systems appliances. This includes full Excel Integration (EMIL) for publishing and
subscribing to market data. BCC Group also is a development partner to Solace
Systems and was the first partner company distributing Solace Systems technology in
Europe.
BCC Group has developed FMDP based on Solace messaging technology for the
high-speed distribution of bank owned, permission controlled financial market data.
BCC Group also developed the first Bloomberg Feed Handler with full EMRS control
connected to Solace devices serving customers based all over the world. BCC Group
maintains strong development partner relationships with Solace Systems,
Morningstar, Bloomberg, Thomson Reuters, Software AG, Microsoft, mongoDB, ORC
Software and others.
FMDP White Paper | November 2015 Copyright © BCC Group International GmbH & Co. KG 2008 - 2015
16
BCC Group International GmbH & Co. KG
Taunusanlage 1
60329 Frankfurt am Main, Germany
Telephone: +49 69 50 50 60 – 704
Fax: +49 69 50 50 60 – 520
Email: [email protected]
Website: http://bccgi.com