© 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Reactive Oriented Architecture with Grails and Groovy

By Steve Pember CTO, ThirdChannel

@svpember

Welcome everyone to Reactive Oriented Architecture with Grails and Groovy. !My name is Steve, I work for a Boston-based startup called ThirdChannel, and I’ve been drinking coffee literally all day. Let’s talk about Architecture

We’ve got a journey ahead of us folks, over an hour. I’ll stop periodically for questions. Last session of the day, I know it’s a haul. We’re almost there.

THIRDCHANNEL

Agenda:

• 1. Monolithic Apps Will Ruin You • 2. Service Oriented Architecture and Microservices • 3. Reactive Applications • 4. Practical Development (Caveats) • 5. Case Studies (Including 3C)

3

While I will be talking about Reactive Applications with Groovy, there’s a good deal of information surrounding the approach that should be covered first. From a high level, I’d group these points into the following topics: 1. Monolithic Apps are Evil and will Ruin You 2. Service Oriented and Microservice Architecture, specifically their benefits and when you would or wouldn’t want to use them 3.

A Few Questions…

Have you ever...

THIRDCHANNEL 5

Gone through a major refactoring of your application?

Have you ever...

THIRDCHANNEL 6

Gone on an ‘Archeological Expedition’ in the codebase to add a new feature?

Have you ever...

THIRDCHANNEL 7

Thought “Why are there so many #@$!% tables in this database?!”

Have you ever been distraught at the number of joins needed to fetch a batch of records?

Have you ever...

THIRDCHANNEL 8

Dreaded executing the test suite?

-Who here has worked on a major project where the unit tests took longer than 15 minutes to run? -30? - Did you ever notice that developers tend to ignore running the tests when that happens? How was build stability? - On the flip side, has anyone worked on a major project that had effectively NO tests?

These are all symptoms of a Monolithic application. If you laughed, I’m assuming that the answers were a ‘yes’.

Monoliths

-A monolithic app is: Single, logical executable or unit, often comprised of 3 components for a web app: the View layer (CSS/HTML/JS), the middle tier (contains business logic) and the data storage. -For example, here’s an example of an e-comm app: product browsing, catalog management, order processing, shopping cart

THIRDCHANNEL

Monoliths Feel Natural

• Contains Everything

12

If you’re just starting out with building an application, a Monolith feels natural !The Monolith Contains all application features and all logic necessary for handling a request within a single process.

THIRDCHANNEL 13

Your developers can generally run ANNNND test the entire app at once, on their laptops

THIRDCHANNEL

Monoliths Feel Natural

• Contains Everything • Minimum Viable Product can be reached quickly

14

— Perhaps their most attractive feature, is that a development team - even a very small one - can reach the Minimum Viable Product very quickly.

THIRDCHANNEL

Monoliths Feel Natural

• Contains Everything • Minimum Viable Product can be reached quickly • Single Deployment

15

There’s only a single artifact to deploy, which makes deployment and DevOps incredibly easy! … (he says sarcastically)

Monolithic apps scale horizontally by deploying whole new instances of the monolith, probably behind a load balancer. !This is wasteful, as we scale the entire application rather than just the portions that require it. For example -> if we’re receiving a flood of requests to the Order Processing, we don’t need to scale the whole thing !

However…

I generally regard these single, logical units as an Evil piece of software. Like I said, it feels like the right thing to do… and they tempt you with how easy it is to in the beginning. Why?

You start out strong, adding controllers here, domain objects, etc. You can be blinded by early gains toward your first release that you become shortsighted and aren’t thinking towards the future.

As those Domains grow, you start adding relations all over the place. Can you really look at a class or schema diagram like this and really think, “Yeah, that’s awesome. Nailed it.”? -btw, this was the largest I could find on Google images. I’m sure we’ve all seen the massive versions of this that cover multiple pages, taped to the wall. - If you ever have to tape to the wall, done something wrong

The Complexity of your application will soon become enormous. !Any gains you think you may have made at the beginning of the project will not continue, particularly once you start to attract users. ..And grow your team.

Scaling Development is hard

Scaling Development is hard work

The ability to add new features, fix bugs, or resolve technical debt cannot scale with the size of the application. Throwing more developers at it also will not help, they’ll just get in each others’ way. - One of the best passive-aggressive statements I’ve seen - Passive aggressively scrawled on the wall: You can’t make a baby in 1 month with 9 women.

-And trying to refactor anything? Forget it. -I’ve seen it happen plenty of times: Your company will start out by creating a team to reimplement a feature or rebuild an application. Sounds good? Eventually, this team will start taking too long. Soon enough, the managers will be screaming for updates or new features on the old app. So, you end up with *two* teams, one maintaining the old app, while the other continues the rebuild AANNND maintains feature parity with the old system.

THIRDCHANNEL 25

As the size of your codebase increases, the computational complexity will exponentially increase and will have adverse effects on maintenance. This has been known for some time; there’s been papers written on the subject. Although it’s difficult to accurately measure.

It can also just be downright miserable. Has anyone here ever opened their project, looked at the maze of controllers or domains - that seem to just explode out at you when opening the folder - and… just give an exasperated sigh? Toiling away on a project which is always growing… always broken… and never seems to get fixed. You work on a small piece all day and nothing ever seems to get better? …This can have an adverse effect on developer morale and motivation.

This is such a misguided architecture, it makes you wonder how exactly we collectively ended up here.

I blame it largely at the feet of the large, popular web frameworks, e.g. Rails, Django, Grails. !Don’t get me wrong, they have their uses. And yes, I see the … irony… in complaining about them while at a framework conference !

But they’re touted as if they’re the magic cure for building your app or product; that your company should be based entirely within a framework. You hear folks say, “Oh, what are you using in your startup?” “Why, We’re a Ruby on Rails shop, obviously!”. !I argue that your framework choice is largely irrelevant (unless it’s Grails ;)! ). When people ask you “what technology are you using?” the answer should be something like “Ugh, well, that’s difficult”.

Architecture Choice is More Important Than Any

Framework

*And we all know that only JVM languages should even be considered

Architecture Choice is More Important Than Any

Framework*

THIRDCHANNEL

Questions so far?

• 1. Steve really hates Monolithic apps !

• 2. Monolith Development will not scale !

• 3. Architecture Choice > Frameworks

32

So, to recap: !!Any questions?

Shattering the Monolith

-Let’s take our anecdotal design, an e-commerce store.

With SOA, one would break up each feature into an individual component or service, and setup communication between each service over some direct service calls. Eg SOAP or REST. -Note that these calls are traditionally Synchronous over http, so it’s imperative that each node return as quickly as possible

This architecture keeps keeps the individual features easier to manage and keep track of, which reduces overall complexity dramatically as we can examine each service in isolation, rather than one big code base

However, before I go into why that’s so important: SOA, specifically, is not without its own issues. !First, having that web of interconnected services can grow out of control. Having to configure service nodes so that each knows where the others are located in the network can be cumbersome to manage.

THIRDCHANNEL 38

ESB

Furthermore, anecdotally, organizations tend to treat SOA differently and have different definitions for what, exactly, SOA is. -In many cases, Enterprise Service Buses (or ESBs) have arisen to deal with complicated traffic. An ESB acts as a piece of middleware that ‘intelligently’ routes traffic to the appropriate server. Which sounds useful, except This adds an additional layer of complexity to an already complex approach, especially if you’re rolling your own ESB. -Often, I’ve seen clients that would hook up several monoliths together and call it SOA -It’s terrible when these all come together.

The Rise of Microservices

Given those pain points, I think that leads right into talking about Microservices, which is a buzzword you certainly couldn’t have escaped hewing about after that keynote last night. And every other talk here at SpringOne

THIRDCHANNEL

Microservices

• Distillation of SOA

40

So what exactly is a Microservice architecture? !I firmly believe Microservices were born out of a desire to be more specific when we talk about Service Oriented Architecture. !

If you look at what people are writing about the approach, you can see that it is clearly an attempt to be more focused about using services. And, certainly, how we manage the teams working on them !

THIRDCHANNEL

Microservices

• Distillation of SOA • Single Context

42

A Microservice should be concerned with only one ‘context’. This helps to avoid having each service become a monolith on its own

Back to this example, here we’ve split up each of the specific components into their own bounded context. !I say “bounded context” because that’s an important metric. You may hear folks say “ A microservice needs to be under 1000ish total lines of code”. Or something like that. !Which is Ridiculous. A microservice should surround a particular bounded context within your application. Meaning that it is the responsible for a particular section of contiguous business logic… it’s the authoritative source on one ‘thing’.

THIRDCHANNEL

Microservices

• Distillation of SOA • Single Context • Smart Services / Dumb Pipes

44

-A Microservice Architecture emphasizes that communication between services be ‘dumb’. -As opposed to ESBs, which can contain complex routing and even business logic - Rather, keep all business logic contained within the relevant Service

Back to this example. Each service should communicate with each other via ‘dumb’ (as in, single-minded) methods. For example, we could use straight up HTTP to talk between services. Be “of the web”. Use Http Resources (HATEOAS anyone?) between services

Or, Use Message Brokers

Or, Preferably - in my opinion - Communicate between your services using an asynchronous Message Broker and Message Queues !There are several excellent ones that you can use. We could send quite a bit of time just talking about RabbitMQ

Message Brokers generally follow this pattern: -Messages are routed through an EXCHANGE in to one or many queues. -Messages are then plucked off the queue by an attached, waiting Consumer. Or, the first available if multiple are attached.

THIRDCHANNEL 48

RABBIT MQ

-we’ve broken the features into individual applications and spun up one node per each -connected the users’ cart to a queue, and our order processing into another queue !- the quickest win in doing just this is that now our application can place items in a users cart and take orders

*asynchronously*. Imagine a flurry of users ordering from you and you’re unable to take their money fast enough.

- here, the order is placed on the queue and your users go back to browsing or purchasing more. This frees up request threads in the front-end for other users as you’re not blocking on the billing component, which may require expensive credit card operations.

THIRDCHANNEL

Microservices

• Distillation of SOA • Single Context • Smart Services / Dumb Pipes • RESTful Communications

50

In a distributed system. One of the most ridiculous things you can do is try and share state across services. Thus a core component of a microservices architecture is to NEVER share state

Attempting to keep shared state across distributed services WILL lead to madness. You can certainly have mutable state *within* a service, but trying to sync that is just ridiculous. As we were talking about before with the simple communication strategy: make REST a part of it

THIRDCHANNEL

Microservices

• Distillation of SOA • Single Context • Smart Services / Dumb Pipes • RESTful Communications • Dedicated, Cross-Purpose Teams

52

Teams in charge of a service should come from a variety of disciplines. Engineering, QA, Art & Design, Project Management, perhaps a deployment engineer, etc. !(If anyone laughs : Yes, I mixed Star Trek and Star Wars references.

Benefits of SOA & Microservices

Before I … eventually… get into some of the … caveats with this architecture, Let me tempt you with why it’s great !Why would I want to do this?

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases

55

It’s dramatically easier to reason about smaller codebases

Each engineer should be able to understand and hold the majority of the service within their mind,. !Thus, when a defect occurs, it’s easier to know exactly where the problem code lies. When a new feature is needed, it’s easier to know exactly where to build and how it will affect other components.

!Maintenance and adding new features now becomes vastly easier. You’ll find that iterating on a particular service is much quicker, and it’s easier to context shift when and if it’s time to work on another service.

Which results in rapid long-term development

In addition, by focusing our developers into teams responsible for each service, this keeps them more focused and out of each others way. Which, again, is a very common problem with monolithic apps. If you’re working on a monolith, how often are you getting merge conflicts?

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases • Easy To Scale / Deploy / Destroy

59

-we’ve broken the features into individual applications and spun up one node per each -connected the users’ cart to a queue, and our order processing into another queue

-suppose the cart and order processing start experiencing heavy load - programatically spin up new instances to handle it - Does everyone remember Andy Glover’s talk from yesterday? He had a thread on how rapidly iterating is the key to beating out your competitors.. and I

think, right here, is exactly what he was talking about… -

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases • Easy To Scale / Destroy /Deploy • Easy To Refactor

62

Both in terms of adding new features, or removing old ones !If we need to refactor a particular feature in a service… say, the inventory check when adding a product to a cart, I can do so without impacting the rest of the application. !Further, if we decide to totally revamp our billing service (say we want to try out Ratpack), we can develop the new service quickly (as we’re starting with something new) and slap it on the queue, again without directly impacting or interrupting the others. !A/B Testing: We could also compare our refactored billing service by having both old and new listen on the queue, then compare how effective the two billing services are.

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases • Easy To Scale / Destroy /Deploy • Easy To Refactor • Right Tool for the Job

64

-not every service needs use the same languages, frameworks - With an SOA or Microservice approach, each service can be constructed with the technologies that best suit its context and functionality.

-If the main product browse / web interface is best suited for a standard web framework, like Grails… perfect -What about the shopping cart? Maybe it should be built with a Redis Data source? No sweat -How about the Order placement? Let’s use some Spring Boot, with some Spring Integration on a Postgres database.

Plus, I firmly believe that this helps to improve the morale of your developers. - When it’s time for a new service, people get generally excited. -Get to work with something a bit new -Design a new project -There’s contentment in that

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases • Easy To Scale / Destroy /Deploy • Easy To Refactor • Right Tool for the Job • History Is Isolated

67

Because each service is kept as a separate product, it should ideally have it’s own repository. Therefore, one nice added effect is that each product now has it’s own isolated revision history

THIRDCHANNEL

Yay for Microservices

• It’s easier to reason about smaller codebases • Easy To Scale / Destroy /Deploy • Easy To Refactor • Right Tool for the Job • History Is Isolated • Mitigates Conway’s Law

68

Has anyone heard of Conway’s law?

THIRDCHANNEL 69

“organizations which design systems ... are constrained to produce designs which are copies of the

communication structures of these organizations”

Conway’s Law

-Melvin Conway, 1968 http://en.wikipedia.org/wiki/Conway's_law

It’s a tongue-in-cheek quote from programmer Melvin Conway from the 60’s. His law states that the structure of a computer system reflects the social structure of the company that made it. While something of a joke, it does have some truth to it.

THIRDCHANNEL 70

DBAs

Engineers

UX / Design

QAFront End Service

Internal Tools Services

Order Management Service

Slide of different groups of people (Designers, Front-End Engineers, Back-end engineers, DBAs, Dev Ops, QA) (color coded) vs image of different service teams with different color coded people in them (e.g. front end team with a couple designers and no DBAs; order-management system with 2 DBAs and no designers)

Save Money!

!Probably the biggest advantage of all of this is that in the end, it should save you time, headaches and money !

THIRDCHANNEL

Save Money

Fewer Resources &

More Efficient Developers

72

If any of you perked up or are skeptical, I don’t blame you. Mentioned this to a coworker, and he said “All I see is that you want to pay Amazon a bunch more money for these additional machines”

Borrowed this slide from Martin Fowler’s website !As demand on the application grows, I can scale just the components that are needed, rather than needing to replicate the whole monolith as demand grows

In the beginning, this may not seem like a huge advantage. !However, consider that as your application becomes ever larger and more complex… more resources are going to be needed to power the entire thing. !Which results in more machines and resources that may just be there to patch over the the small bounded context of your application which is truly demanding the attention. If I have 6 x demand per usual on the billing system only, I argue that it should cheaper to add 6 small billing system nodes than to have to deploy our monolith 6 times across more expensive machines

THIRDCHANNEL

Save Money

Fewer Resources

& More Efficient Developers

75

!

Smaller Codebases Are Faster CodeBases

Developers are More Expensive than Machines

Spending Less Money on Developers and Machines

Use that money to buy developers beer

developers love beer

There are certainly some downsides of the micro service approach, but we’ll get into that later. !Generally though, I hope you agree that SOA and Microservices do a decent job of fighting the Monolith…

… but we can do even better. Squeeze out even more power from our resources.

THIRDCHANNEL

Questions?

• 1. Microservices are the natural enemy of Monoliths • 2. Small Code is Fast Code • 3. Save Money

82

CAP Theorem

Next, I want to diver a little and discuss something called the CAP Theorem

THIRDCHANNEL

Cap Theorem

• (C)onsistency • (A)vailability • (P)artition Tolerance

84

THIRDCHANNEL

Cap Theorem

• (C)onsistency • (A)vailability • (P)artition Tolerance

!

• Pick 2

85

And the CAP Theorem says that in a distributed application, your system can only fulfill 2 of those attributes

THIRDCHANNEL

Cap Theorem

• (C)onsistency • (A)vailability • (P)artition Tolerance

!

• Pick 2 • Theorem From Paper Which Proved ‘Brewer’s Conjecture’

86

Originally called Brewer’s Conjecture In 2012, this paper contained a formal proof of the Conjecture and it graduated to full-on theorem. !

THIRDCHANNEL

Cap Theorem

• (C)onsistency • (A)vailability • (P)artition Tolerance

!

• Pick 2 • Theorem From Paper Which Proved ‘Brewer’s Conjecture’ • Not Fixed / CAP Choices Can be at Service Level

88

Web Apps -> A & P Money -> C

Sliding Scale

THIRDCHANNEL

Cap Theorem

• (C)onsistency • (A)vailability • (P)artition Tolerance

!

• Pick 2 • Theorem From Paper Which Proved ‘Brewer’s Conjecture’ • Not Fixed / CAP Choices Can be at Service Level • Origin of Eventual Consistency

91

A little bit of trivia -> this paper also introduces the term the term “Eventual Consistency” originated

THIRDCHANNEL

Eventual Consistency

• High Consistent System: After Update, all components see change

• Eventually Consistent System: After Update, eventually all components will see change

92

… as it propagates throughout the system. And that’s ok, for most things. Again, for anything dealing with Money, you’re going to want a highly consistent system.

Architecture Choice is More Important Than Any

Framework

Now, knowing all of that, I reiterate: Architecture choice…

If We accept that Microservices are Good… how do we build each Service?

How can we squeeze even more power out of our applications? How can we further increase the maintenance advantage of microservices? How can we further reduce our resource costs?

Reactive Microservices are the ideal Web

Architecture

And so I believe that the ideal architecture for the web is one based on Reactive Applications built on distributed microservices -> not merely SOA. A distributed Microservice architecture, where each service follows the Reactive approach

Going Reactive

But I’ve been stalling… So, when I say ‘Reactive’, what exactly do I mean?

The term ‘Reactive’ is a bit of a buzzword…

Popularized by a company called Typesafe. They are the maintainers of the Play and Akka Frameworks, !!

4 Reactive Principles

THIRDCHANNEL

4 Reactive Principles

• 1. Event-Driven

100

Principle 1: Event-Driven

Asynchronous Message Passing

Essentially, a Reactive system should be built around the concept of Asynchronous Message Passing. Take advantage of the asynchronous features of your language or framework, and…

-Communication within the system should be done via done via asynchronous immutable events, rather than long procedural code. -This naturally promotes highly decoupled code. Sender and recipient can be constructed without having to know - or care - about implementation details of the others. - I think this is a very important point… and is perhaps the most important facet of Reactive programming as the others may come naturally from adopting this single step

NEVER BLOCK

In other words, though…

THIRDCHANNEL 104

Suppose we had an E-commerce app which included a service which created and processed orders. In this version, we use several additional service collaborators to do things like verifying that we have inventory, charge the user, and send emails if successful or not. Each of those service calls could take a while (checking inventory, sending email, money), which blocks the thread while we wait for each service call. This slows down the request not only for the end user but for any rampaging hordes trying to access our app

VS

<slow>Now, compare this to…

THIRDCHANNEL 106

Here’s that same method. Where did the rest of the code go? First, note the ‘notify’ with a String-based event

THIRDCHANNEL 107

On order:created, we execute the first services job Then we check the inventory for holding, and broadcast a new event based on those results

THIRDCHANNEL 108

The moneyService will bill the client if the inventory is good, While the email service will respond either way. !-Now, some of might be saying, “But Steve, that code is spread across 3 different slides!”. While, true, which of these would you rather use? Which version do you think is more decoupled? Which would be easier to test? !-(Imagine having to mock all these services as collaborators, versus simply creating events)

THIRDCHANNEL

4 Reactive Principles

• 1. Event-Driven • 2. Scalable

109

Principle 2. A Reactive Application must be scalable

-The system should be able to efficiently grow based on the demand placed on it. Should be able to replicate easily on or across multiple nodes. !Dynamic scaling is an optimal use of resources: can scale up quickly when needed, but can scale down when not needed (save $) !

Application Topology will Change

As we build, we may not know what the application topology will be at the start… and it may change in the future. Design your systems such that they: <next>

1. Minimize Contention on Resources

-if one thread or process is blocking access to a service or another resource, than other threads, connections, etc will have to wait. This reduces the CAP availability, will require more resources (e.g. more machines), and will anger the user if they notice the slowdown. -horrible contention will have a direct monetary cost as you scramble to scale your resources and keep the end user happy -this will start to come as a natural emergence of adopting Asynchronous message passing

2. Maximize Locality of Reference

Include -> part 1. here, build up a list Maximize locality of reference. This is a very complicated topic…

So much so that I couldn’t think of a snappy image for it. !-If data is accessed frequently, it’s a good candidate for caching (Temporal) -Keep related data close together (Spatial). Minimum number of joins to build a particular record !- This takes on a different, sinister, twist when working with Microservices, which I’ll get into in a bit

THIRDCHANNEL

4 Reactive Principles

• 1. Event-Driven • 2. Scalable • 3. Resilient

116

Principle # 3, Be Resilient

Embrace Failure

In fact, *embrace* that catastrophic failures will most assuredly occur.

Have the mindset that at any given moment, your code is going to break. Think to yourself: How coupled is my code? How dependent on other services, domain objects, third party services, etc is my application? -How many of your systems are held together by metaphorical duct tape? What happens if any one of those go down? The worst thing you can do is know that catastrophe is coming or how it *could* come and to do nothing about it.

-A Reactive application is resilient to failure. If one service node breaks down, the others should be able to take up the slack. If one feature goes down, the others should still operate. -Your system should be able to suffer damage and still operate. -For example, in the ecomm example, if the order placement feature goes down.. while your team is scrambling to fix, the end user should still be able to browse the products and add items to the cart. -They should just see a message that says “Sorry, order placement will be right back”… -or better yet, take the order, send a note to the customer, and process the orders later once the system is back online

Independent Things Fail Independently

Your resilient mantra should be “Independent things fail independently”

Occurs naturally with Async and Microservices

Design your systems and services to be as loosely coupled as possible. Using Async messages and microservices helps this behavior to arise naturally

THIRDCHANNEL

4 Reactive Principles

• 1. Event-Driven • 2. Scalable • 3. Resilient • 4. Responsive

122

Principle #4, Responsive

Your application, in general, should respond to user interaction as quickly as possible. Thus, each service should handle events quickly. !A more ‘responsive’ application is a pleasing experience to your end user. The faster your application responds to their input, the less time they sit staring at your app, and the happier they’ll be. !TODO: Add Amazon delay -> cost Several years ago, an engineer with Amazon named Greg Linden revealed that the company ran experiment where they intentionally slowed down site performance - 100ms delay resulted in 1 percent drop in sales. !

In Other Words…

The 4 rules of Reactive Applications are… -Sorry, I watched Fight Club a week or so ago. As a result, I’ve got a bit of an anti consumerist streak going. … he says while giving his presentation on a Mac laptop

1. NEVER BLOCK !

2. SHARE NO STATE !

3. INDEPENDENT THINGS FAIL INDEPENDENTLY

!

4. BE FAST

No Single ‘Correct ‘ Reactive Architecture

Rather, Reactive is a mindset or a state of being. It’s a goal that you need to orient your thinking around when you are designing or architecting your individual service nodes and your distributed architecture

The people at Typesafe wrote up this thinking into something they call the ‘Reactive Manifesto’. After this talk, if you like what I’m saying, feel free to go and sign it.

THIRDCHANNEL 130

“An Application must be Reactive from top to Bottom”

- The Reactive Manifesto

So, going back to the original e-commerce example… - how to go reactive?

First of course, is to break up our application into individual components. We could build an API to communicate between services via synchronous HTTP…

Never Block!

But remember, never block!

Route State-less Messages Asynchronously

Each service should send and respond via the message broker by passing messages asynchronously. (rule #1) Each message should be entirely stateless (rule #2).

- A Reactive system will emerge from Reactive services. - Once you reach the point where messages are asynchronously bouncing around your system, you’re going to

sit back and just whistle. Unless you’re me. I can’t whistle.

THIRDCHANNEL

Reactive Questions?

• Reactive Is a Mindset !

• 4 Rules !

• Reactive Services Emerge into a Reactive Distributed App

136

Practical Development

Anyway, onto a more interesting topic.

THIRDCHANNEL

Development

• Compose Services From libraries

138

by nature, working with micro services have an amazing amount of code reuse. !-an organization will tend to standardize the deployments and internals of new services. This is generally a good thing, but can lead to a great amount of code reuse and boilerplate

THIRDCHANNEL 139

Slide of legos? - Thus, extract the common code into sets of shared, tested, libraries - Compose every service out of these shared libraries, so that when working on each service you only have to reason about the custom code

THIRDCHANNEL

Development

• Compose Services From libraries • Architect / Design Team

140

As mentioned earlier, need an architect to keep everyone on task and following the Reactive microservice approach.

This can be an authoritative committee as well, if that’s too much power in one person’s hands But keep in mind that everyone in that committee is *going* to have an opinion

THIRDCHANNEL

Development

• Compose Services From libraries • Architect / Design Team • Empower your Teams / Make them Responsible

143

THIRDCHANNEL 144

THIRDCHANNEL

Development

• Compose Services From libraries • Architect / Design Team • Empower your Teams / Make them Responsible • Embrace Eventual Consistency

145

Embrace Eventual Consistency

The concept that when a change is made to an object within the distributed system, that each service or database may not be consistent with that change, but they eventually will be. This can be a difficult thing for people to accept, but for most objects I argue that’s OK. !-Amazon misses an order

THIRDCHANNEL

Development

• Compose Services From libraries • Architect / Design Team • Empower your Teams / Make them Responsible • Embrace Eventual Consistency • Be worried about Reference Locality

147

Seriously, be concerned about Locality of Reference

Each service should be an authority on some section of your data In general, other services should query the authoritative server when necessary (Thus maintaining spatial locality) - However, each separate service will need to be aware of the domain objects. Each Service will have a different understanding - or ‘projection’ - into the particular Domain

In an ideal environment, each system would be completely separate. -However, there will always be some conceptual overlap. For example, let’s say we have a system where <describe system>. -Red lines are anchors between data localities - Here, users’ uuid is a shared natural key which acts as each services’ anchor to understanding the concept of a user. In our main user service

understands a plethora of personal information - Our order history system also understands the user, but simply maintains the order history joined with the user id

Same chart from before, but another service, email. This service also conceptualizes the user, but with a different subset of information. Namely, the email address. - Email address is stored in the user information service, but is *also* used extensively by the email service. Which one should be the authoritative

source? The email service uses it more (and thus has higher Temporal locality), but email is a key piece of information which should be managed by the user auth system.

- Tough Choice - One Compromise is to (unfortunately) synchronize the email address across multiple services - Or, communication service has no concept of User’s email address, but simply blindly sends emails out. The email addresses involved must be placed

into each message received by the Communication service

Practical Development with Groovy

The Groovy / Grails / Spring / general JVM world is filled with very interesting libraries and tools. It’s an exciting time to be building these scalable, distributed applications.

THIRDCHANNEL 152

Akka - I’m not smart enough to understand Actors Actors -> relatively small implementation. Each receives messages in their ‘mailbox’ or message queue. Walled off from you in terms of configuration and deployment: you access actors via a proxy. This means that actors can be easily distributed across machines for dramatic increase in computational power.

THIRDCHANNEL 153

Reactor (which totally needs a logo)

Reactor is my favorite. It’s the easiest to use with the most straightforward applicability. A Reactor instance is a wrapper around one of several different Dispatcher types (e.g. a standard Event Loop, an LMAX Disrupter Ring Buffer). A reactor acts like a central dispatcher within your service node

The folks at Netflix have been kind enough to give us ReactiveX, a suite of various reactive tools for a variety of (primarily JVM) languages. !

<sample rxGroovy Code> The rxGroovy (or rxJava) libraries give you a tool called an Observable Sequence with which it’s possible to assemble reactive applications. !We can do simple asynchronous list processing or transformations…

…

or a publish subscribe approach I believe it’s an extraordinarily useful tool for working with Groovy sequences in an asynchronous fashion.

Plus the documentation is well done, with some excellent diagrams showing event flow

And, with Grails specifically, we have the built in Async Support !In my opinion, the two most useful currently are the Promise support and Asynchronous Request Handling

Promises! !EXpand on that promise a bit

Slightly more complicated example, using promise chaining. A few things to note: - use of async keyword - merge -> GOOORRMM! -

Asynchronous request Handling, very simple case

Remember to verify that your servlet version set to 3.0 in BuildConfig.groovy

But Be Careful

It’s not all sunshine and roses (cupcakes and snowflakes?), however. There are areas of difficulty with Reactive microservices, both in deciding whether or not to build them and the practical development.

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone

165

THIRDCHANNEL 166

-A Word press blog that your parents are building, a local restaurant site, or any general informational page… -Probably not a good candidate for this approach - Now, for your startup… you may be saying “I’m not yet a huge internet giant. This seems hard”. - How much more difficult will it be when you eventually realize that a distributed approach would be a huge help? -

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone • Initial Investment is LARGE

167

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone • Initial Investment is LARGE • Initially Very Difficult

168

This approach is an investment…

THIRDCHANNEL 169

An investment of both time and money. At first, a micro service architecture may look crazy, particularly perhaps, to your bosses. It’ll pay off dividends in the end, but that initial investment can be cost prohibitive.

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone • Initial Investment is LARGE • Initially Very Difficult • Make sure Everyone’s on Board

170

!If someone on your team doesn’t agree or doesn’t Understand the microservices approach, it’s possible that they may create faulty services

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone • Initial Investment is LARGE • Initially Very Difficult • Make sure Everyone’s on Board • Can Lead to Team Animosity

172

Because teams are responsible for a service, it can lead to an us-vs-them mentality. If a service is broken, the teams relying on that service can began to fall back to a ‘Well I can’t do any work because the User Management service is down”. Encourage teams to cooperate. Cycle team members to other groups periodically.

THIRDCHANNEL

Caveats

• Not The Right Fit For Everyone • Initial Investment is LARGE • Initially Very Difficult • Make sure Everyone’s on Board • Can Lead to Team Animosity • Conway’s Law

174

Be ever watchful for Conway’s Law. Ensure that your teams are working together when designing their services… otherwise you’ll find that certain services are tailor-made to communicate more efficiently and directly with others.

Case Studies

In the end, though, it’s all worth it. !Some of the largest companies out there are demonstrating to the world that this approach is ideal for the web.

First up: our good friends Netflix !-make heavy use of AWS -replicate bundles of services as demand dictates -Netflix is big on resiliency. In order to test their resiliency, they use a custom tool called ‘Chaos Monkey’

Every weekday between 9am and 5pm (a.k.a. “Work Hours), Netflix unleashes this little adorable little guy into their infrastructure, where he goes to work randomly destroying various services.

And the Chaos Monkey was just the first. They have a whole army of monkeys that do things like: - simulating high latency and service degradation - clean up unused resources - monitor health checks to detect and alert for unhealthy instances

These services need to prove their strength and stand up to the simian army. Or rather, this allows Netflix to find weaknesses and bugs in their services

Slide showing Rx - repos

Netflix <3’s Groovy

- Groovy is used extensively throughout Netflix’s services - Whatever insane amount of the world’s web traffic goes through their services? That’s touching Groovy code - the next time someone tells you that Groovy’s too young, or we need Java, or some other excuse, just point to Netflix

…Twitter is another large one

They had one of the largest, most monolithic Rails deployments in the world. -Engineers needed to pull in Global Experts to get anything done on the codebase -Spent more time on “Archeological Digs” or “Whale Hunting expeditions” to track down bugs or learn how

things worked than on developing new features

Turning point came during 2010 World Cup. Flurry of tweets brought twitter to its knees. Service was unavailable, engineers worked through the night Debated putting a vuvuzella sound effect here.

Change was needed

https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

They shattered their monolithic application into multiple individual applications.. The size of the circles represent the number of deployment nodes of that particular application service -I know I just spoke about Rabbit, but Twitter developed their own tool called Finagle which handles asynchronous communication between nodes

Each application had their own small, determined, dedicated teams.

-they switched from the Ruby Virtual Machine to the JVM. -Embraced Event-based programming from the bottom up Went with Netty (as opposed to something like NodeJS), which is an NIO technology for the JVM

Monolithic: 200 - 300 requests / sec / machine

!

Reactive: 10-20k request / sec / machine

During the 2010 World Cup, Twitter hit a record of 3283 TPS

During the 2010 World Cup, Twitter reached a record (at the time) of 3283 tweets per second before the service collapsed.

Now: average 5700 TPS Dynamically scale to 143,199 TPS

Now, they average 5700 tweets per second… but their systems are set up so that they can dramatically scale up on demand. The current record is 143,199 tweets per second. That’s not a hard number… that’s just the most they’ve received. They could theoretically handle more, it just hasn’t happened yet.

With 5x - 12x fewer nodes

And they’ve done all of this with 5 to 12 times fewer service nodes. !Not bad, yeah?

//That’s kind of mind blowing, right? Not Bad, eh? !-> Incidentally, JVM advocates and Rails detractors love to talk about how Twitter moved away from Rails to Java. And while that’s true that Java would give better performance, the Architecture choice was vastly more important

How we’re using it

THIRDCHANNEL 196

About ThirdChannel

- I should spend a bit of time explaining my company ThirdChannel. Normally, I might save it, but in order to fully understand the context it’s good to have a bit of background on us

This is a slide from a sales deck, I apologize for the marketing speak But basically, the problem we’re trying to solve… <explain 3c>

THIRDCHANNEL

About 3C

• In-Store Product Awareness !

• Observational + Sales Data !

• Product Intelligence & Analytics

198

Another sales slide, but this gives a flavor of what our current feature set is(or 2 months ago current, actually), and how it looks on mobile

We Need Many Services

… or rather, we have many internal bounded contexts that I feel can be utilized as discrete services

(Show slides of services, adding one by one) First, we have the web interface. -Contains the Marketing Site, plus the basic structure of the internally facing site.

!We started with gathering observational data (rather than empirical sales data), and this second service is still our most used context (beyond, of course, the front-end service). This service is responsible for receiving and providing a query-able interface for the data we gather.

To gather all of that data, we use Agents. However, each Customer could have several dozen to several hundred agents, so we need services to both help our staff with Agent Recruitment, and Agent Management (e.g. which stores they’re currently assigned to). The checkin server and I believe also the web interface query the agent management service to determine which stores an agent is both shown and allowed to provide data for.

And we do pay those agents pretty well… so we need a service to track payments per some payment schedule. Each Customer may pay agents differently, so that logic needs to be handled. Every two weeks

And all of that, so far, is mostly just for Agents! !Notification Service: sends emails and tracks internal messages within the app.

And then services for pulling sales data…

as well as a bunch of additional services for aggregating data into different query able formats

And we’re still going

-And we still have more to do - This is just what we’ve managed to work towards this year. Is each one perfect? Absolutely not. We’re iterating on each one of those services - plus a few

other very minor ones - constantly…. and rapidly, without impacting other services or areas of the business - if we go back one slide, we’re mostly working on the analytics services right now … in addition to revamping our main front end - and we’re not impacting

the large agent management team we have at all

Our Lessons Learned

During this process there have certainly been stumbling blocks, though

#1

The absolute First Thing…

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format

211

Agree on a Messaging format !(Image of strife) Our discussions weren’t actually like that. But, you know. not far off. What data type are you going to use xml? json? How about format? Something already established, like HAL? How much data should be sent? Should every service be able to customize the response it wants? What about when broadcasting general informational updates

#2

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format • #2 Build Communication Libraries

214

The very second thing that should be built is a shared library (or libraries) for facilitating communications that utilizes the pre-mentioned messaging formats

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format • #2 Build Communication Libraries • Maintain API Documentation

215

Next, make sure that - for each service - you maintain proper API documentation

This documentation about each service should cover it’s externally facing concerns. !Describe which events it emits and what sort of data it sends out during each event. Describe how to communicate directly with the service, if at all. Do you talk to it directly via HTTP? Does the service respond to RPC calls over the message queue? What sort of input does it expect?

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format • #2 Build Communication Libraries • Maintain API Documentation • Remember Conway’s Law

217

Keep communication up between service teams… ESPECIALLY when you’re just starting out. !When these communication patterns are still in the early stages, it’s very important that everyone in your entire team understand how it will work !I’d recommend a meeting once or twice a week for service team leads to meet and discuss things like “So I’m planning on emitting this event. Any thoughts one data you all might need?”, or “Hey, can you update the shopping cart service to also send this data along with event Y?”

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format • #2 Build Communication Libraries • Maintain API Documentation • Remember Conway’s Law • Always Think of Data Locality

218

THIRDCHANNEL

Lessons Learned

• #1 Agree on a Messaging Format • #2 Build Communication Libraries • Maintain API Documentation • Remember Conway’s Law • Always Think of Data Locality • Consider Capturing & Logging All Events / Auditing

220

In Summary

Architecture Choice is More Important Than Any

Framework

Reactive Microservices are the ideal Web Architecture

The ideal framework for the web is a Reactive Microservices approach

If you want to work with Reactive Microservices, you need to adopt the mindset.

Beware CAP and Data Locality

Finally… we should all strive to be Like Netflix.

Thank YOU!

And with that, I say THANK YOU.

THIRDCHANNEL

Any Questions?:

• 1. Monolithic Apps will Ruin you • 2. Service Oriented Architecture and Microservices • 3. Reactive Applications • 4. Practical Development (Caveats) • 5. Case Studies (including 3C)

228

Steve Pember CTO, ThirdChannel

@svpember steve@thirdchannel.com

In conclusion, this is what I attempted to go over today. Any questions?

Complexity (pipes): http://www.flickr.com/photos/bhophoto/384574407/ Simple Pipeline: http://tierracos.com/our-companies/tierra-pipeline/

Highway: http://farnorthdallas.advocatemag.com/wp-content/uploads/2013/04/882349_500402383328407_539732406_o.jpg Telephone Exchange Operator: http://fineartamerica.com/featured/telephone-exchange-1920s-granger.html

People in Queue: http://www.guardian.co.uk/money/2010/mar/23/dole-queue-jobseekers-online Cookie Monster: http://muppet.wikia.com/wiki/Cookie_Monster_Through_the_Years

Too Many Cooks: http://www.ecommercesystems.com/featured-articles/cooks-kitchen-driving-ecommerce-business/ Resuscitate: http://www.dailymail.co.uk/health/article-2034160/Do-resuscitate-Theyre-fateful-words-meaning-doctors-wont-try-save-you-collapse-

hospital.html Twitter Logo & app chart: https://blog.twitter.com/engineering

Modern Server Farm: http://bookriot.com/2013/03/26/book-less-libraries-and-other-contemporary-realities/ Grey World: http://upload.wikimedia.org/wikipedia/commons/d/d0/BlankMap-World-1ce.png

Monolith: http://filmessaysandarticles.wordpress.com/ Star Trek: http://www.gifbin.com/984064

Reactive Manifesto: http://www.reactivemanifesto.org/ Cheetah: http://www.livescience.com/21944-usain-bolt-vs-cheetah-animal-olympics.html

Buzzword Bingo: http://www.amsterdamadblog.com/columns/buzzword-bingo/ Businessman meditating: http://www.huffingtonpost.co.uk/2013/07/09/meditation-successful-businessmen_n_3567694.html

Chaos Monkey: http://luckyrobot.com/netflix-chaos-monkey-keeps-movies-streaming/ Darth Vader: http://www.businessinsider.com/hillary-clinton-darth-vader-poll-2014-7

Turtles: http://people.wcsu.edu/pinout/herpetology/tcarolinei/29366343.EasternBoxTurtle2.jpg, http://www.huffingtonpost.com/2013/04/26/24-tiny-turtles-who-need-a-reality-check_n_3134793.html,

http://www.ecology.com/2012/05/23/world-turtle-day/ “Maintenance Rocks”: http://www.prlog.org/12205104-grace-hill-inc-declares-september-maintenance-appreciation-month.html

NFL Ref: http://www.mprnews.org/story/2012/09/03/news/nfl-referees “Say No to Sharing”: http://www.cultofandroid.com/15519/sprint-starts-say-no-to-sharing-campaign-against-shared-data-plans/

“Dominoes” https://www.flickr.com/photos/louish/5611657857/sizes/l/in/photostream/ “First Day on the Internet Kid” http://knowyourmeme.com/photos/239112-first-day-on-the-internet-kid

“Tortoise and the Hare”: http://www.theguardian.com/sustainable-business/social-enterprise-blog/2014/feb/10/week “Star Trek”: http://keyepisodes.blogspot.com/2013/10/star-trek-next-generation.html

“Darryl Issa” http://nypost.com/2014/03/05/shouting-match-erupts-during-house-hearing-on-irs-scandal/ “Fight Club” http://pyxurz.blogspot.com/2012/05/fight-club-page-4-of-7.html

“Woman looking at phone” http://remedybail.com/blog/page/4 “Guy with metal detector” http://beachurbanity.blogspot.com/2010/04/beach-metal-detector.html

“Happy dog” http://sydneespetgrooming.com/blog/ “Trekking in the desert”: http://www.wildmorocco.com/

“morpheus”: http://www.desktopexchange.net/movie-pictures/matrix-wallpapers/ “Monoliths and Microservices” http://martinfowler.com/articles/microservices.html#CharacteristicsOfAMicroserviceArchitecture

“Calvin fighting” : http://commonsenseatheism.com/?p=2214 “Sample Restaurant” : http://www.letseat.at/

“Solar panels”: http://prepare-and-protect.net/2014/02/solar-panels/

Image Credits