dynamic routing at 1 million messages per second with spring integration
DESCRIPTION
This is the deck for the presentation that John Davies (@jtdavies / [email protected]) and I gave for a webinar (http://spring.io/blog/2013/10/16/nov-19th-webinar-dynamic-routing-at-1-million-per-second-with-spring-integration) on the topic of using Spring Integration and C24-technology to process highly specialized data such as that typical of financial services solutions at high speed.TRANSCRIPT
Dynamic Routing at 1 Million Messages per Second with Spring Integration
John Davies, CEO, Incept 5 @jtdavies !Josh Long, Spring Developer Advocate, Pivotal @starbuxman
!2
SPRING INTEGRATION 101
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
what is “EAI”?
EAI encompasses approaches, methodologies, standards, and technologies allowing very diverse but important systems to share information, processes, and behavior in support of the core business.
-David Linthicum
“ ”
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
what is “EAI”?§ file transfer § shared database § remote procedure call § messaging
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration
§ At the core, an embedded Message Bus § Also, an Application Integration
Framework § Support Enterprise Integration Patterns*
via the Spring programming model § Build on the Spring Platform
n (n-1)2
A
C
B
D
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration§Payload can be any object §Header values are stored in a read-only java.util.Map<K,V> §Similar to JMS messages, email mime envelopes, XMPP messages, etc.
public interface Message<T> { MessageHeaders getHeaders(); T getPayload(); }
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration§Decouples producers from consumers § provide extension points for interceptors §messages may be queued and buffered
<channel id="async-p2p"> <dispatcher task-executor=”simpleAsyncThreadPool" /> <queue capacity=“10” /> </channel>
producer consumermessage channel receive(Message)send(Message)
point to point
publish-subscribe<publish-subscribe-channel id="async-pubsub” task-executor="someThreadPool" />
producer message channel
receive(Message)
send(Message)
consumer
consumer
consumer
!!!{ !
!!!{ !
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration
transformers: convert payloads and modify headers
filter: discard messages based on test
router: determine next channel based on message and test
splitter: generate multiple messages from one
aggregator: assemble a single message from multiple messages
service activator: delegates processing to a backend service call
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration
<channel id = “customerChannel” /> !<jms:outbound-channel-adapter channel = “customerChannel” destination-name=“stocks” connection-factory = “connectionFactory” /> !
producer
jms:outbound-adapter
send(Message) jmsTemplate.send(javax.jms.Message)ActiveMQ
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration
<channel id = “fileChannel” /> !<file:inbound-channel-adapter channel = "fileChannel " directory= "#{systemProperties['user.home']}/Desktop/in "> <int:poller fixed-rate="1000"/> </file:inbound-channel-adapter> ! file:inbound-channel-adapter
new File(directory).list() Message<File> mf = fileChannel.receive() consumerDocumentDocumentDocument
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Spring Integration
!MessageChannel channel = ... Message<Customer> customerMessage = MessageBuilder.withPayload(customer) .setHeader("customer-id", 10) .build(); channel.send(customerMessage);
!13
DEFINING PERFORMANCE
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
We need a good safety margin
• If we can process 200k/sec and we get a peek of 400k for an hour (e.g. Christmas, Thanksgiving, breaking news etc.) • We have a backlog of 200k/sec for 3600 seconds, if the volume drops to
150k/sec it will take 4 hours to catch up
• And we’ll need 360 GB of RAM to queue it! And that’s just 500 bytes/msg !
• RabbitMQ was used at a large “fruit” vendor • Queue sizes were critical in the planing of the deployment !
• Even working at 500k/sec and a peak of 550k for 30 minutes dropping to 400k • 15 minute’s delay and 45 GB of RAM !
• We need “REALTIME”
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
• In order to do any work on a message we need to parse it • For XML that’s a SAX parser, DOM or Java Binding (JAXB, JIBX etc.) !
• If you’re going to store, search or deliver messages these are the basic steps… • Parse the message or part of the message you need
• Query and route the message or…
• Query to search the message
• - Possibly serialise and de-serialise the message
• - Possibly enrich or transform the message
• Write the message out (if it’s been bound to Java) !
• All this takes time :-(
Where’s the problem?
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Java Binding
• Keeping the example simple a CSV generates something like this…
public class Row extends ComplexDataObject {! private static final long serialVersionUID = 1L;! private String name;! private String cardNumber;! private String expiryDate;! private double amount;! private boolean isamountSet;! private String currency;! private Date transactionDate;! private double commission;! private boolean iscommissionSet;! private long vendorID;! private boolean isvendorIDSet;! private String country;
Name,Card Number,Expiry Date,Amount,Currency,Transaction Date,Commission,Vendor ID,Country!Stephen Hawkins,4325-6486-3757-2674,10/06,100,GBP,26-09-2006,2,14988603,UK!Tim Berners-Lee,4724-7345-4725-7833,11/07,258,USD,21-12-2006,5,15688632,UK!Bill Clinton,4924-7264-1264-8532,04/09,1250.6,USD,13-09-2006,15,66846035,US!Angela Merkel,4457-4356-0087-0107,05/08,350,EUR,13-11-2006,7,93440252,DE!Richard Branson,4724-7345-4725-7833,11/06,250,USD,22-02-2006,5,14988103,UK!Tim Cook,4924-7264-1264-8532,04/09,12250,USD,16-09-2006,1.3,67434435,US!Alan Turing,4325-6486-3757-2674,10/06,50,GBP,23-02-2006,1,14119663,UK
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Message Parsing
• Taking a message with about 50-60 attributes and parsing them so that they can be available to Spring Integration/Batch is not exactly complex !
• But providing the ability to manage the changes and dozens of standards starts to add complexity • Things change over time, we need to manage that change too
• We’d also like a consistent API so Java-Binding is still ideal !
• Two relatively simple examples from the Telco and Financial Services industry are RADIUS (rfc-2865) and FIX (from FPL) • Others include ASN.1 & ISO-8583
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Radius (rfc-2865)
• Remote Authentication Dial In User Service (RADIUS) • 77 pages of binary spec...
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Modelling in C24 first
•The RADIUS standard...
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
What we can now do
• The spec may say that bit 13 of a 32 bit field represents the presence of a field ABC (later in the message) • Programatically we can test bit 5 with a mask 0x00002000
• Using the generated code we can simply call isAbcSet() !
• It may say that bits 4-6 represent the version ID • Programatically we can mask it and then shift it...
• mask 0x00000070
• shift >> 4
• Using the generated code we can simply call getVersionId() !
• We now have a nice API that hides the complexity of the binary implementation
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Bits and Bytes to…
• We can now go from this… !
jd-server:Radius TestData jdavies$ hexdump -C radius.dat 00000000 01 02 00 74 ea d5 7c 62 1f d0 f6 fe a3 bf 36 4c |...t..|b......6L| 00000010 35 25 e5 8c 1a 17 00 00 28 af 01 11 32 33 34 34 |5%......(...2344| 00000020 35 37 30 36 32 37 38 38 35 33 36 01 11 32 33 34 |57062788536..234| 00000030 31 35 39 30 36 32 35 38 38 35 33 36 1f 11 33 35 |159062588536..35| 00000040 33 34 32 31 30 32 30 39 34 35 35 36 38 5e 0e 34 |3421020945568^.4| 00000050 34 37 30 30 34 31 38 38 36 37 33 1a 0d 00 00 28 |47004188673....(| 00000060 af 08 07 32 33 34 33 35 06 06 00 00 00 02 37 06 |...23435......7.| 00000070 4e 97 57 a8 |N.W.| 00000074
!
• To being able to use it in Spring • Or Mule, Fuse, Camel etc…
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
To Spring Integration...
• Test bits and binary data in native Spring…
!
• <filter input-channel="filter-message-channel" output-channel="process-message-channel" ref=”payload” method=”isAbcSet"/>
!
• <filter input-channel="filter-message-channel" output-channel="process-message-channel" expression="payload.versionId == 5"/>
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
A good start but…
• We now have a pretty cool API and can use Spring but it’s still pretty slow as it generates LOTS of objects !
• With our telco standards binary to Java binding was “slow” due to the number of objects, we got around 20k/sec/core • Fine for most purposes and with an 8 core machine potentially good
for 100k/sec but we wanted better !
• We needed a better solution, it wasn’t the parsing but the complexity of the objects we were creating and parsing into
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
ByteBuffer
• Java 1.4 added java.nio.ByteBuffer
• Basically a wrapper for a byte[]
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Lazy parsing
• Rather than parse all of the elements from the message every time we retrieve the elements only when we need them • Similar to comparing a DOM with a SAX parser
• We assume that we will only need to filter/sort/query on a limited number of fields so we save a lot of redundant parsing !
• Performance goes from ~20k/sec (50µs per message) to ~1 million/sec (1µS per message), some 50 times faster • Even parsing the entire message is significantly faster
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Here’s where it gets interesting...
• So we now have a Java object with a ByteBuffer holding the message data and dozens of get() methods to get the content • Message call = new Message(data);
• call.getDuration();
• This works pretty fast, plus the JIT compiler kicks in after 10,000 iterations and optimises the method
• Now the bottleneck moves to the SpEL queries • <filter input-channel="filter-message-channel" output-channel="process-message-channel"
expression="payload.duration lt 0.1"/>
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Reflection
• What’s happening under the hood is reflection, very powerful but sadly still rather “slow” • expression="payload.duration lt 0.1"
• Turns into something like...
!
!
!
• This adds about 700nS to each message • A few of these and we’ve more than halved the performance
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
The Java Compiler to the rescue!
• New from Java 1.6: ToolProvider.getSystemJavaCompiler();
• We can create a generic accessor for double values...
!
!
• Can now write a class on the fly that implements this method for the getter we want e.g. getDuration()
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Running it...
• Load up the compiled class (or byte-code) and run it
!
!
• Note however that the first two lines (above) are done only once, outside of the loop
!
• The result is “native” performance as if it was code • Which of course it is
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Back to spring...
• SpEL expressions are currently interpreted
!
• There is some basic caching • java.lang.reflect.Field
• java.lang.reflect.Method objects are cached once discovered and reused on subsequent evaluations !
• But overall they are “slow” (for what we need) • We need to look at avoiding reflection by compiling the parsed
expression into a class
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
SpEL compiler
• Currently proposed by Andy Clement (Spring guru) is a SpEL compiler • With the proposed changes SpEL has a Mixed Mode Interpreter
(MMI) system in place
• Mixed means it mixes the current interpreter with a real expression compiler • When an expression is evaluated X number of times SpEL compiles it to byte-code and
uses that going forward
• User does nothing, evaluations just accelerate
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Does it make a difference?
• Property access:foo.bar.boo
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Does it make a difference?
• Method invocation:hello()
!
!
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Advantages - Performance
• We can now parse, compare and filter in around 1 µSec (1 million/sec) per core • Make decisions on what you want early on and drop unwanted
messages
• Provide “on-the-fly” aggregation
• Around 90-95% less CPU per message - $€£¥₹ !
• Disadvantages • Less error checking
• Hardware vendors make less money
• Machine rooms become uncomfortably cold
• Power bills go down
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Other Advantages
• Messages in memory are binary encoded in ByteBuffers and so occupy a fraction of the size of a bound Java object • Typically around 1/12 of the size, using only 8% of the memory !
• We can queue around 10 times more messages per unit of memory !
• We get around 1000% capacity advantage for in-memory grids - GigaSpaces, Hazelcast, GemFire, Coherence !
• We can provide custom serialisation of the ByteBuffer for all of the above - boosting node to node replication
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
So - In a nutshell
• We’ve gone from 20k per second to around 800k • 40 times faster, sometimes better !
• We’ve gone reduced memory footprint by around 12 times • Meaning you can store 12 times more data in your grid
• Or use 1/12th of the hardware/memory !
• You can route this with Spring at “native” JIT-compiled Java code speeds • So far no other framework can offer this level of efficiency, dynamic
flexibility and performance
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
The Future
• SpEL to native Java compilation to be included in Spring release • A few issue remain to be resolved in the short term but early versions
look good !
• Full support for standards like ASN.1 • This is a standard wire-level implementation option for ISO-20022
• ASN.1 supports multiple encoding rules, Basic, XML, Canonical, Packed etc. !
• XML messages encoded as packed ASN.1 (PER) would be almost as performant as raw binary • Complex XML as packed binary in a ByteBuffer
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
More Spring
• Read more on the tools we used... !
• Spring XD - http://projects.spring.io/spring-xd/
!
• Spring Integration - http://projects.spring.io/spring-integration/
!
• Spring Batch - http://projects.spring.io/spring-batch/
!
• Spring AMQP - http://projects.spring.io/spring-amqp/
Copyright © 2013 Incept5 Ltd. http://www.incept5.com
Thank you!
John Davies
Twitter: @jtdavies
LinkedIn: http://linkedin.com/in/jdavies/
E-Mail: [email protected]
C24: http://www.c24.biz
+Josh Long
Twitter: @starbuxman
GitHub: http:// github.com/joshlong
E-Mail: [email protected]
Spring & Spring Integration: http://spring.io