a business leaders guide to event-driven architecture · event-driven architecture this ebook...

EVENT-DRIVEN ARCHITECTURE

This eBook defines EDA, highlighting the advantages and challenges, suggesting best use cases, and

describing the internal structure in general terms with the goal of enabling better decision-making

regarding deployment. Not only will you be able to better determine where to apply EDA, but also how.

A BUSINESS LEADERS GUIDE TO

A BUSINESS LEADERS GUIDE TO EVENT-DRIVEN ARCHITECTURE2

04.

CONTENTSINTRODUCTION

CHAPTER 1: WHAT IS EVENT-DRIVEN ARCHITECTURE?Introducing the fundamental characteristics of Event Driven Architecture (EDA), when and where it is best applied, and how it interacts well with the processing of microservices in containers.

CHAPTER 2: COMPONENTS OF EVENT-DRIVEN ARCHITECTUREThe layers of this architecture are examined separately in this chapter, beginning with an exhaustive list of the various components active within, along with helpful descriptions of each.

CHAPTER 3: EVENT-DRIVEN ARCHITECTURE TOPOLOGIESSee how events flow from capture through processing to triggered response or responses. This is the first step toward actual event processing.

CHAPTER 4: EVENT PROCESSING APPROACHES INEVENT-DRIVEN ARCHITECTURELearn how events are assigned to the appropriate processors and accessed by the intended consumers.

CHAPTER 5: ADVANTAGES OF EVENT-DRIVEN ARCHITECTUREDiscover how EDA delivers us to a world in which everything must be delivered upon demand.

CHAPTER 6: WHEN TO USE EVENT-DRIVEN ARCHITECTURESMore and more of today’s applications, especially those that facilitate communication between var-ious devices, are best served in asynchronous fashion with components loosely coupled with others. EDA powerfully facilitates efficiencies from the systems they run on.

CHAPTER 7: EDA AND MICROSERVICESWhen using microservices in containers every component stands alone, separate, and loosely coupled from others. This is where EDA leads you into an environment of continuous improvement through continuous deployment (CI/CD).

05.

09.

18.

23.

26.

29.

33.


CHAPTER 8: DISADVANTAGES OF EVENT-DRIVENARCHITECTUREAnticipating when and how challenging issues may arise, and documenting guidance to respond to

those issues is a problem, but the payoff is the kind of interoperability needed for tomorrow’s systems.

CHAPTER 9: OVERCOMING CHALLENGES OF EVENT-DRIVENARCHITECTURESSince the components involved are not all on the same systems there’s no roadmap to follow. The best hedge against this will be careful, comprehensive governance and documentation. EDA could just as easily signify Extremely Disciplined Architecture.

CHAPTER 10: EVOLVING YOUR SYSTEM BY IMPLEMENTINGImplementing EDA represents far more than simply adopting a new architecture. It requires making a paradigm shift from a history of request-driven to a brand new event-driven practice. This is no mean feat. You’ll find yourself reminding yourself to think differently.

CHAPTER 11: LEVERAGING A PARTNER FOR EDA SUCCESSEngage a team of experts who work together to help you achieve your EDA implementation. Yes, it

takes a village.

CONTENTS38.

42.

46.

50.


Architectures adapt to advances in technology. From the very beginning of the Information Age, architectures have been request-driven, that is, a user makes a request and the system responds.

Most recently we’ve seen computing systems go beyond person-to-machine (P2M) interaction to extensive machine-to-machine (M2M) interaction. In the age of the Internet of Things (IoT) sensors communicate events back to a central system that may instruct messages to be sent, lights or other indicators to go on or off, bridges to be raised, street lamps to light streets, and much more.

As M2M becomes a larger proportion of online interactions than P2M it becomes necessary to accommodate the idea that events will be the triggers of action rather than requests.

This has given rise to the development of Event-Driven-Architecture (EDA), an advanced structure forperceiving the occurrence of an event, interpreting and processing that event, determine what action oractions must be taken, and triggering those actions.

Interoperability is the core enabler of IoT, Smart Home, Industrial Internet of Things (IIoT) and automation. Systems created by different manufacturers and software developers must be able to receive notification of specific events from each other and be able to reliably interpret and evaluate those events to determine what response is required. Then they must trigger that response without operator intervention. EDA wasdeveloped to serve that need.

EDA is also designed to deliver excellent resilience and fault-tolerance. It is highly granular and looselycoupled similar to the structure of microservices in containers that have recently overcome monolithicdevelopment limitations.

This eBook defines EDA, highlighting the advantages and challenges, suggesting best use cases, and describing the internal structure in general terms with the goal of enabling better decision-making regardingdeployment. Not only will you be able to better determine where to apply EDA, but also how.

INTRODUCTION


WHAT ISEVENT-DRIVENARCHITECTURE

Chapter 1:


EVENT-DRIVEN ARCHITECTURE EXPLAINED

Ideal for real-time product creation, highly resilient thanks to loose coupling of services and wide distribution of processes, event-driven architecture (EDA) is becoming more and more popular among today’s software developers.

But what is EDA? It’s not a new language. It is, however, a new programming approach. Perhaps a feweveryday examples of what it is and is not will best serve to illustrate.

EVENT-DRIVEN ARCHITECTURE EXAMPLE

You’re in the lobby of a building but you need to be on the 20th floor. You press the button that calls theelevator which then travels down to meet you, opens its doors, welcomes you in, and then takes you to the floor of your choice where it opens up and lets you out.

Pressing the button produced the event. The event listener, the elevator’s operational system, processed the request sending it to the event consumer, the elevator car itself which then processed the trip from the ground floor to the 20th. Nothing at all happened until the event of pushing the button happened.

Note that the button never knew anything about what the elevator did or whether it was successful or not. Nor did it care!

Traditional Request Driven Architecture Example

You’re driving with your young family to a far-off destination. Not long into the trip one of your children asks, “Are we there yet?” You answer “no.”

That same child asks “Are we there yet?” again a few minutes later and again you answer “no.”

The next time the child asks you simply don’t answer. In fact you don’t answer the next several dozen times the child asks.

Finally you reach your destination and turn around waiting for the question to be repeated one last time. The child quietly mumbles, “Are we there yet?” and this time you answer “Yes!”

As opposed to the event of pushing the button triggering the action, your child was polling you periodically to see if the desired result was available yet. You continued processing until your destination was reached at which point your response to the poll was positive.


CHARACTERISTICS OF EVENT-DRIVEN

The first advantage of Event-Driven Architecture (EDA) becomes immediately clear. The polling, in this case, was annoying. In systems the polling is simply expensive and burdensome, consuming bandwidth and cycles usually with no result. In fact, EDA service provider Zapier estimates that 98.5% of polling processes are valueless.

One of the most popular modes of EDA is pub/sub in which a service is published making it available. Event producers subscribe to this service and trigger events as needed. Response to order happens in real-time providing a significant competitive advantage.

Other advantages include:

• The event producer, event notification, event router, event listeners, event consumers, and event proces-sors are all loosely coupled and widely distributed so the failure of one does not cause others to fail.

• In fact, the event router acts as an elastic buffer storing events during workload surges or failure-driven delays. Each request is then processed as soon as its listener and consumers are back online demonstrating how fault-tolerant this programming approach is.

• Because the event router is centralized and also stores information on every event it is fairly easy to trace back and identify where a process may have gone wrong. This facilitates auditing, policy definition and enforcement, and coordination between event producer and consumer services.

• No custom code is ever required to poll, filter, or route requests. Development is accelerated thanks to the event router.

• EDA enables tremendous horizontal scaling. A single event may trigger many responses resulting in a wide variety of results. The creation of real-time products is significantly enhanced.

• Most processes are asynchronous allowing for potential latencies elsewhere.

WHEN SHOULD YOU USE EVENT-DRIVENARCHITECTURES?

EDA has found application in a wide variety of use cases including real-time analytics, fraud detection,sentiment analysis, business process monitoring, real-time personalization of user experience, healthcare monitoring, Medicaid Management Information Systems, IoT applications, trading floors, online shipping,and more.


Placing an order on an eCommerce website in an event that triggers many activities occurring in parallel. Products are picked. Shipping containers are prepared. Bills of lading and other documentation are printed. An order confirmation is sent. Then products are shipped, receipt is confirmed, invoicing occurs followed by receipt of payment. After the event of placement of the order the rest may easily be completely automated.

Where are Event-Driven Architectures Used?

The past few years have seen several EDA services introduced primarily to the consumer audience. Oneexample is the IFTTT.com service, an acronym which stands for “If THIS then THAT.” Users authenticate a wide variety of internet-based services they use. The “THIS” becomes an event on one service that triggers the “THAT” to occur on the target service. Simple applications including operating lights, door locks, and more on voice command, having all household lights and appliances shut down when a specific time of night is reached. Newer services alert emergency services when sounds like glass breaking, fire alarms going off, and other potential emergencies occur.

Moving more toward the business market is Zapier which functions similarly but offers a wider variety of available business-oriented triggers and responders.

EVENT DRIVEN ARCHITECTURE AND MICROSERVICES

Event Driven Architecutre (EDA) aligns well with the increasing popular use of microservice architectures. Events never travel, they just occur. Everything else needs to be portable.

The more each service can stand alone, the more resilient and fault-tolerant the system is.The economies of this programming approach create the likelihood of greater adoption in the field.

Inner Workings

Having discussed the why and what of EDA, let’s next drill down into the components of the architecture to begin developing an understanding of how to make it work.

A BUSINESS LEADERS GUIDE TO EVENT-DRIVEN ARCHITECTURE9 A BUSINESS LEADERS GUIDE TO EVENT-DRIVEN ARCHITECTURE9

COMPONENTS OF EVENT-DRIVENARCHITECTURE

Chapter 2:


Making a request and getting a response is a common occurrence both in programming and in the real world. You walk up to an elevator and press the call button. The elevator shows up a few seconds later, the doors open, and you step in. You press the button for the floor you’re going to and the elevator takes you to that floor. Two button presses, both requests, and two responses from the elevator. Simple.

In an equally simple digital example, you click on a weblink which issues a request that takes your browser to the desired website.

As new applications emerge, especially in the Internet of Things (IoT) space, it becomes clearer all the time that the interactive, interoperable world won’t wait for requests.

“Things” in the IoT tend to fall into two categories, sensors that detect things happening, and switches that enable responses to make other things happen. For example, a thermometer may detect that the temperature has risen above 80° F. in a given space. Exceeding that threshold is the event. Turning on the air conditioning is the result.

FEATURES OF EVENT-DRIVEN ARCHITECTURES

First developed almost twenty years ago, Event-Driven Architecture (EDA) creates an environment in which there’s no waiting for requests. Instead, they detect when any of many various items changes state sufficiently to warrant a response. Then they perform that response. The architecture is asynchronous and distributed. Events may occur while resources are unavailable to respond to them.


EDA makes provision for storing such event notifications until resources become available. It is also highlydecoupled. None of the event producers or consumers, or the processors and data transport that connect them are aware of each other and remain so even after the transaction is completed.

In our air conditioning example, the air conditioner may want to know whether to turn it on low, medium, or high based on the current temperature. The notification of the event can include specific data about that event which may be useful in determining how to respond.

EDA is not a specific software language, it is an architecture, an approach to programming, a model for more efficient solution for the capture, communication, processing, and persistence of events. As such, event-driven applications may really be created in any programming language.

EDA processes are highly decoupled and can be scaled independently, which makes ita good option for modern, distributed and cloud-enabled applications that arehorizontally scalable and resilient to failure.

Fundamentally, EDA detects something significant and meaningful happening and distributes usefulinformation to all humans and automated systems that would be interested in having it. Those interested parties may or may not take action in the form of launching other services, commencing specific business processes, or obtaining further information.

Extremely Loose Coupling

EDA is extremely loosely coupled, and highly distributed. The producer of any event only knows the event occurred, with no knowledge of subsequent processing, or even the identity or existence of the interested parties. This loose coupling enables great flexibility but makes tracing errors in a dynamic multipath event network challenging. EDA is best used for asynchronous workflows.

Choice of Topologies – Mediator

Use mediator topology when orchestrating multiple steps within an event through a central mediator.

Choice of Topologies – Broker

Use broker topology to concatenate events without using a central mediator.


EVENT PROCESSING STYLES

Events may be processed in any of several styles. Mature EDA systems often combine more than one.

SIMPLE EVENTPROCESSING (SEP)

Typically used to takelatency and cost out of

business processes.

COMPLEX EVENTPROCESSING (CEP)

Is used when evaluatingmultiple events and taking

action as necessary.

EVENT STREAMPROCESSING (ESP)Stream event processing detects ordinary events as well assignificant and meaningful ones.

PUBLISH/SUBSCRIBE(PUB/SUB) PROCESSINGPub/sub simply denotes publisherspushing messages to subscribers.

Simple Event Processing

Typically used to take latency and cost out of business processes, simple event processing simply initiates action further down the application stream whenever a significant and meaningful change of state occurs in any hardware or software component of the system.

Stream Event Processing

Stream event processing detects ordinary events as well as significant and meaningful ones, screening them to determine significance and streaming them downstream to information subscribers. This aids in enabling in-time decision-making.

Complex Event Processing

Complex event processing (CEP) is used when evaluating multiple events and taking action as necessary. The events involved need not be of the same type or even occur at the same time. CEP uses far more sophisticated interpreters, pattern definitions, and correlation techniques. Correlation may be casual, temporal or spatial. It is typically used to identify and act on business anomalies, threats, and opportunities.


Publish/Subscribe (Pub/Sub) Processing

Pub/sub simply denotes publishers pushing messages to subscribers. In EDA, pub/sub provides instant event notifications for distributed applications. Given the extremely loose decoupling of EDA this is especiallyuseful in distributing event information to many separate systems.

EVENT-DRIVEN ARCHITECTURE FLOW LAYERS

Since EDA is a software development architecture it exists as a series of flow layers made up of components each of which plays an important part of application design. The sequence of an EDA event flow consists of four fundamental layers, each of which contains multiple components:

Event Channels

An Event Channel is a messaging backbone that transports events in a standard format between EventGenerators, Event Processors, and downstream subscribers.

Event Processing

Actions taken by event processing are based upon evaluation of each event against established processing rules. Possible actions include invocation of a specific service, launching of a business process, publication of the event to subscribers, generation of another event, or simple storage of the event.

Event-Driven Downstream Activity

Events may initiate a variety of activities downstream. Activities such as commencing a service or business process may be pushed by the event processing engine while subscribers may pull event publications as needed. Possible subscribers include applications, data warehouses, automated agents, active business processes, dashboards, or humans.

Event-Driven Architecture Flow Components

Although it has been in use for almost two decades there are still many varying descriptions, definitions, and nomenclature. In the following components list great effort has been made to include as many different names for each component as could be found.

What things are called is not nearly as important as how they work together to enable software to veryeffectively respond to a wide variety of events happening internally, externally, and in the “real” world.

These components are presented in as close to the sequence in which they are involved as possible.


Event

In common programming parlance, the basic attributes of an event include time, source, key, header, metadata and payload. In terms of EDA, an event is simply a significant and meaningful change of state in an object and any specific data that may be relevant to it. That data is often referred to as the event payload.

The object need not be aware and usually is not aware of any of the other components in the EDA system. Put simply, it just does what it does.

Users may be the source of an event as in when they press a key or click a mouse. Sensors may detect an event in the physical world and generate an event as a result. The system itself may generate events, for example when software is loaded. An event may signify a problem, an opportunity, crossing of a threshold or some other deviation from normal. Significant and meaningful events must be specified and evaluated in business terms rather than data or application terms.

Event Producer (Generator)

Every event is generated from a source. The source might be an application, data store, service, businessprocess, transmitter, sensor, or collaboration tool (IM, email). An ordinary event may be evaluated fornotability by an event preprocessor (router, filter), resulting in the generation of a new notable event.

Because of the variety of event generators, not all events will be generated in the required format for event processing. In those cases, the events need to be transformed to the required (enterprise standard)3 format prior to being deposited in the event channel.

Event Emitter (Agents)

Event Emitters detect, gather, and transfer events. They, too, don’t know anything about event consumers, their existence or how events are processed which contributes to keeping components loosely coupled.

Event Notification

An event happens when a significant change in state occurs for any system hardware or software. The system then sends a message to tell other parts of the system that the event has happened. The event itself and the event notification are often mistakenly referred to interchangeably, which is incorrect. The event notification is a result of the event.

The event notification is configured in two parts. The event header provides a name for the event, a time stamp and an indication of what type of event it is. The event body includes additional details about the event that will be useful further along.


To maintain low coupling, the event notification does not anticipate any response.

From a DevOps perspective it’s worthwhile to note that while the event notification is relatively easy to set up, it can become difficult to trace in the event of program failure. The event notification flows through various systems each of which is independent from others. With no fixed workflow to trace, debugging and modification can be challenging.

Event Handler

An Event Handler is a software routine which handles the occurrence of an event.

Event Loop

The Event loop manages interaction between events and event handlers.

Event Carried State Transfer

To reduce system latency an event notification includes all details required to process the event. As it travels between systems, each maintains a copy of this data. The latency reduction occurs when the system doesn’t need to make remote calls for information as it makes updates of its own copy. This also enables rapidreconstruction and reprocessing of an event.

The hazard is that this will create different copies of the same data which can lead to data inconsistency across the system.

Event Store

The Event Store is a database containing all stored events for future use when necessary. Events arepublished to an Event Stream or Message Broker which allows subscribing consumers to access and process them as needed.

Event Sourcing

Event Sourcing stores every change in state as an event in the Event Store which adds significant resilience as any event can be rebuilt at any time in the future. This has the effect of making the Event Store the principal source of truth. Event Sourcing adds strong audit capability but cannot be depended upon as a sole solution as dependence upon other systems may make rebuilding difficult.


Event Queues

The flow of every event begins when a client sends an event to an event queue. This is, in turn, used totransport each event to the event mediator.

Event Mediator

When an event requires that multiple steps be taken some level of orchestration is required to send additional asynchronous events to various event channels. The Event Mediator provides that orchestration.

Event Channels

An Event Channel is a messaging backbone that transports events in a standard format between EventGenerators, Event Processors, and downstream subscribers.

Event Processors

Actions taken by event processing are based upon evaluation of each event against established processing rules. Possible actions include invocation of a specific service, launching of a business process, publication of the event to subscribers, generation of another event, or simple storage of the event.

Event Consumer (Sinks, Subscribers)

Events may initiate a variety of activities downstream. Activities such as commencing a service or business process may be pushed by the event processing engine while subscribers may pull event publications as needed. Possible event consumers or subscribers include applications, data warehouses, automated agents, active business processes, dashboards, or humans.

Event Consumers may choose from a wide variety of responses to each event or may choose to not respond to a given event or store that event for later processing.


UNDERSTANDING EDA IS JUST THE BEGINNING

As you’ve read through these various processes, layers, and components you’ve traced the flow of events through an event-driven architecture. EDA enables systems to respond to events occurring in other systems through extremely loose coupling, as discussed. But this loose coupling also brings a concomitant challenge in that there is no one established workflow that can be traced in an effort to resolve system challenges.

Troubleshooting will undergo transformation, but that’s a natural follow-on to the profound development change embodied in EDA.

Next let’s explore how these many components are assembled into either of two operating topologies.

A BUSINESS LEADERS GUIDE TO EVENT-DRIVEN ARCHITECTURE18 A BUSINESS LEADERS GUIDE TO EVENT-DRIVEN ARCHITECTURE18

EVENT-DRIVENARCHITECTURE

TOPOLOGIES

Chapter 3:


Event-Driven Architecture (EDA) creates an environment in which there’s no waiting for requests.

Response processes are launched when an item changes state. An action or several actions are executed in response. Key advantages of EDA are that it is both asynchronous and distributed. Since events may trigger when resources are not available to respond to them, EDA also provides storage for such event notifications until resources become available in the form of an event queue.

Another advantage is that EDA is highly decoupled. None of the event producers or consumers are aware of each other and remain so even after the transaction is completed. Since EDA processes are highly decoupled they can be scaled independently, which makes it a good option for modern, distributed and cloud-enabled applications that are horizontally scalable and resilient to failure. Given that EDA is made up of highlydecoupled, single-purpose event processing components that asynchronously receive and process events, it is also highly adaptable and can be used for both small and large, complex applications.

CONTENDING WITH COMPLEXITY

Burglar alarms serve as a useful metaphor here.

Imagine a small jewelry store with one general retail space and a backroom. Very simple space. When aburglar alarm is installed to protect it, sensors are placed on the doors and windows. When a robbery begins, the very first event is either the door being broken open, or a window being smashed. Either of these events sets off the burglar alarm. A simple one-to-one relationship between the sensors and the alarm occurs.

Now imagine the burglar alarm system for Fort Knox. Many more doors and windows. Movement sensors and cameras. Other sensors most of us cannot even imagine. But only one of them must be tripped to create the event of a suspected break-in.

Alarms are only the beginning of the response. Large steel doors throughout the complex drop into place. Open vaults automatically close. All cameras begin recording. Several people are contacted directly and alerted. All the lights go on. Many, many things happen in response to that one event. This requires more sophisticated orchestration of the execution of many responses on many systems none of which is coupled to any other in any meaningful way to the others.

This is a far more complex response appropriate for a much larger facility with greater needs. Both responses are commenced and coordinated using event-driven architecture, but two different ways of managing the response are needed to accommodate complex responses and simple ones.


TWO MAIN EDA TOPOLOGIES

To achieve this, event-driven architecture consists of two main topologies, the mediator and the broker. The mediator topology is commonly used when you need to orchestrate multiple steps within an event through a central mediator, whereas the broker topology is used when you want to chain events and responses together directly without need for a mediator. Because the architecture characteristics and implementation strategies differ between these two EDA topologies, it is important to understand each one to know which is best suited for your particular use case.

Mediator Topology

This event-driven architecture topology is better suited for more complex situations in which multiple steps are required to process an event, thus requiring event processing coordination or orchestration, is mediator topology which consists of four main components: the queue, the mediator, the channels, and the processors.

An event sends a message through a pre-processing event channel to a single event queue which willtransport the event to the mediator thus initiating an event stream that routes events to relevant eventprocessors. Orchestration is performed by the mediator, sending asynchronous events to various event channels that will perform each individual step of the process. The event processors, meanwhile, are listening to the channels and receive the event from the mediator. They then perform the business logic required to process it.

The mediator transforms the initial event it receives generating a processing event that can be received by the processors. This is not to say that the mediator itself performs any business logic. It simply providesprocessing instructions to the processor. All business logic processing for each single event is performed by the event processor containing the required application logic associated with that specific task. No other processors will be involved.


Mediator Topology Example

Often, when an event occurs, several processes must be called upon to execute, and not always at the same time. At Fort Knox many things must happen instantly and some later on. A complex series of time-critical events including alerts to enforcement personnel, shutting of emergency doors, lights coming on and klaxons going off. Follow up reporting and analysis must happen and be confirmed later. This requires orchestration making certain that all responses are properly processed by relevant processors. These event processors receive event notification from the mediator and perform the appropriate business logic. This topology is preferred for more complex event processing.

Broker Topology

Broker topology is best suited when you have a simple event stream that doesn’t require event orchestration. No mediator is required, so it can be centralized. Like an event stream in mediator topology it can follow ei-ther a queue model or lightweight message topics model or a combination of both. The only core components involved are the broker and the processors. The broker component is federated and contains event channels used within the event flow. A far simpler model, but it does support multi-channel communication with each channel receiving its own identifier.

There is no event queue or mediator in broker topology, it depends upon the event processors to obtain each event, publish them, and process another event that announces the end of processing. Preserving the loosely coupled nature of EDA, the tasks all remain asynchronous. When a task ends, a callback is triggered. This pub-lished message is not consumed by any other event, which may leave the application open for future updates.

A breach atFort Knox(event)

Check sensor(event)

Send notification(event)

Trigger alarm(event)

Commence lockdown(event)

Eventqueue

Check sensor(event channel)

Send notification(event channel)

Trigger alarm(event channel)

Commence lockdown(event channel)

Notificationprocessor

Sensorprocessor

Alarmprocessor

Lockdownprocessor

Event Mediator


Broker Topology Example

Simple event streams don’t require orchestration, so the broker topology better fits the need. In our jewelry store example, a breach of any of the security devices, such as breaking a window, creates an event that is sent directly to the broker. The broker communicates the event notification directly to the event processors. Since nothing is asynchronous, no event queue is required either. At the end of each process notification of completion is issued by the event processors.

WHICH EDA TOPOLOGY SHOULD YOU CHOOSE?

While the choice of which EDA topology to use is seldom simple, the general rule is that broker topology will likely be your choice for simple, single events requiring one task as their result.A mediator is required when an event sets off multiple tasks in response thus requiring significant coordina-tion or orchestration.

Thief breaks window

event

Theft processor

event

Alarm processorSensor processor Owner notif.processor

Police notif.processor

Window Broken

Sensor check

event

Notify

event

Activate alarm

EVENT PROCESSINGAPPROACHES IN EVENT-DRIVENARCHITECTURE

Chapter 4:



The flow of an event begins with the event producer sensing an event and creating an event message that flows through event channels to event processing which sends instructions and pertinent data to an event consumer that takes specific actions, including no action, depending on the nature of the event. This allows for real-time interactive processing to support inter-operable scenarios such as the Internet of Things (IoT). As evidence of how loosely coupled these components are, the event producer has no knowledge of theconsumer or consumers, nor the outcome of any given event.

EVENT PROCESSING STYLES AND APPROACHES

In this post we examine the processing of each event. Processing results in producing the correct response to the received event and then sends the activity downstream to the right consumer or consumers finally resulting in the desired outcome. To accommodate the wide variety of possible events and desired outcomes several different processing approaches or styles have emerged. As EDA solutions evolve, they may frequently combine more than one of these:

Simple Event Processing (SEP)

Many events are very basic. For example, the sensor in your car that tells you when your tire pressure varies by more than 2 psi. Detecting that event will trigger a yellow light on your dashboard. Simple.

A simple event occurs when some notable, significant, and meaningful change of state or condition occurs. This makes simple event processing ideal for real-time flow of work coordination as it reduces latency and cost of processes. It simply serves to initiate action further down the application stream.

Simple events do not generate summaries but happen independently from other events. The resulting event message may contain additional data needed to facilitate response.

Complex Event Processing (CEP)

Picture the tumblers of a combination lock. For the lock to open all three tumblers must be properly aligned. Similarly, complex event processing is used when multiple events must take place before action is taken. Each individual event need not be similar to the others, nor even occur at the same time. Some may be ordinary events that don’t themselves require response. Others may be notable, significant, and meaningful.

CEP waits until all criteria are fulfilled before creating an event message to communicate action instructions. To accomplish this, CEP uses far more sophisticated interpreters, pattern definitions, and correlation. CEP is used most often to identify and action business anomalies, threats, and opportunities. It is also used to detect a specific series of events occurring within a continuous stream of incoming events.


Event Stream Processing (ESP)

Event Stream Processing drives real-time flow of information around the enterprise, processing streams of event data with the goal of identifying meaningful patterns, and helping support time-critical decision making. Ordinary as well as notable, significant and meaningful events are written to a log in ESP. Event consumers don’t subscribe, they simply read from any part of the stream any time they choose to join it. Ordinary events are passed downstream through event channels to event consumers. This makes ESP ideal for IoT workloads where decisions need to be made instantly.

ESP flow includes four steps of event processing; Collect, Enhance, Analyze and Dispatch. This flow creates a process in which events detected by the system generate a response using several different services. It detects relationships between multiple events, performs event correlation, establishes event hierarchies, and evaluates other aspects such as causality, membership and timing.

Publish/Subscribe (Pub/Sub) Processing

Think of Pub/Sub as the very familiar act of publishers pushing out messages to their subscribers. As such, pub/sub provides instant event notifications to distributed applications, especially important when sharing with many separate systems all loosely coupled in an EDA.

A pub/sub channel has one input channel which then splits into multiple output channels, one for each sub-scriber. After an event is sent out to all subscribers they only get to access it once, and any new subscribers joining subsequently will not be able to replay it, similar to SMS messaging.

Conclusion

Event processors are critical to the ability of EDA to enable interaction and inter-operability betweenvarious disparate applications. Sensors connecting to the Internet of Things depend upon it to determine what actions need to be taken automatically in response to their event reports.

The variety of event processors available enables rapid response to simple single events, complex multiple event dependencies, as well as selective monitoring where required.


ADVANTAGES OF EVENT DRIVENARCHITECTURE

Chapter 5:



The Event-Driven Architecture pattern is not new. The concept first originated in 2000 which was just prior to the dawn of what can be considered the age of interactivity and interoperability.

Event Driven Architecture EDA departs from the traditional request-driven software architecture in which someone had to take an action specifically designed to obtain a desired outcome from the system, to an event-driven environment in which the fact that something happened, an event, causes appropriate responses to be returned by the system.

THE TOP 4 BENEFITS OF EVENT-DRIVEN ARCHITECTURE

You Use EDA Regularly When You Use GPS

GPS uses EDA. To define through illustration, imagine you’re driving on a highway. Suddenly, your GPS system alerts that it is re-routing. It does so because something happened, an event occurred. In this case, the event was a traffic accident ahead of you on the road you’re driving on. The result of that event has already started becoming backed-up heavy traffic that will delay you if you remain on this road. Instead, your GPS recommends that you switch to a parallel route. It even chooses that route for you and directs you as to how to get there.

That entire process began with a careless driver not looking or signaling before they changed lanes. An event that otherwise had no relation to you, nor was it aware of you. It just happened. Neither was the traffic monitoring system that detected the accident and reported it to your GPS guidance system aware of you. And you weren’t really aware of any of these systems either, at least until now.

You made no request to be shown traffic patterns on the various area highways. In fact, you made no requests at all.

All the systems involved, including you, your GPS, the guidance system, and the traffic monitoring system are very loosely coupled to each other. Barely at all. They all quickly coupled when the event was detected so it could be routed and processed to notify you to get to another route. The traffic monitoring system also coupled with many other systems to provide the same notification.

This is the greatest benefit of EDA. It scales easily providing highly responsive reactions to allmanner of applications and providing access to needed data in real-time to fuel fast, effectivedecision-making.

The Importance of Interoperability

The successful operation of the Internet of Things (IoT) is clearly heavily dependent uponfrictionless interoperability. EDA enables highly divergent, loosely-coupled systems to detect events happening in other systems that require response.

1

2


This, in turn, enables a new level of automation that is highly responsive delivering substantialefficiency increases in delivering the right information to the right systems and people, making fully-informed decisions, and taking decisive action faster than ever before.

The loosely-coupled nature of event-driven architecture systems also facilitates rapid access to large and varied volumes of data generated by IoT devices and stored across various connected net-works. Without the traditional overhead of integrated data access systems, more information, and more robust information, is available faster and to more endpoints than was possible otherwise.

Resilience and fault-tolerance are also enhanced by the underlying nature of the loosely-coupled computing nodes that constitute event driven environments. Should a node fail while processing a particular event, that failure becomes an event that is detected immediately triggering the re-in-stantion of that node to complete the required functions. Since event driven architecture systems also record every activity it is possible for them to recover from failure by replaying recent events to re-establish conditions at an earlier point in time.

Event Driven Archtecture Enables Asynchronous Functionality

The other fundamental characteristic of Event Driven Architecture that is so tremendouslyvaluable is that all functions are completely asynchronous which completely discards the linear nature of traditional software design.

No longer are developers bound by highly specific code that pre-determinesevery action.Instead, the occurrence of an event determines what actions will be taken next. While this may be harder for developers to document, it provides unheralded flexibility and openness in system design.

Also, since EDA systems are self-documenting they are also naturally optimized for the use in real-time analytics. Incorporating artificial intelligence (AI) and machine-learning (ML) technologies will evolve this into self-healing, self-improving performance.

Living in an Everything-On-Demand World

Perhaps the most confounding challenge in using EDA is the difficulty in providing root causeanalysis of any given failure. As event producers and event consumers proliferate it becomes more and more difficult to trace back activities occurring within and between them. Like an interstateautomobile pursuit the most difficult transition occurs when the getaway vehicle reaches andcrosses over the border where the chase must then transition between the first state’s lawenforcement vehicles and the second. This is an ideal place for the getaway car to get away.

To refine our understanding of the advantages of EDA and to put them into context, let’s follow the why and what with when EDA is the right architecture for your specific application.

3

4

WHEN TO USE EVENT-DRIVEN

ARCHITECTURES

Chapter 6:



Ask experts what is needed for the success of the Internet of Things (IoT) now and in the future and one of the responses you will often hear is “interoperability.” This makes sense. The devices themselves never know what they will be asked to interoperate with, who made it, what its protocols are. Like the introduction of printer drivers in the distant past, something is needed to enable any device to connect to any other device.

The past few years have seen rapid changes in many aspects of how we use technologies. Classic mono-lithic applications have given way to orchestrated microservices in containers that are readily transported throughout the “cloud.” In addition to interoperability, we need greater interactivity. The traditional method of having a user issue a request or command has been replaced by the more interactive concept of identifying an event, a change in state sufficient to warrant response or responses from other parts of the system, or of other systems.

A common example of EDA begins when a motorist runs a red light. A camera mounted near the traffic light is triggered by an event, the movement of the automobile beyond the stop line. It then records the event. That recording, along with the time and location data, triggers the registering of a violation and the creation of a summons.

Automated mailing systems mail out those summonses and the motorist receives theirs a few days later. The violation is eventually resolved when payment is issued, another event, but the record of the violation remains for consolidation with previous and future violations to drive the decision to suspend that driver’s license temporarily. Note that the payment began with the violator writing and mailing a check or using an online credit card service that is received by the system that issued the summons. This example integrates interactions between different systems as well as human involvement.

As to when to use Event-Driven Architecture:

EDA is ideal when you want to capture behaviors in the real world as well as the digital, along with pertinent facts that can be used for decision-making.Record of the event is stored for future analysis. This reflects how humans react to events happening around them.

FROM REQUESTS TO EVENTS

According to Cisco Systems, the number of devices connected to the global internet exceeded the number of people using it in 2008 and has continued to increase at an accelerating pace ever since. Where the original interactions conducted across the internet were from people to machine and back to the person, thepresence of tens of thousands more devices produced interactions between machines.


The change from request-driven architecture to event-driven parallels the increase in machine-to-machine (M2M) activity as opposed to the decreased percentage of internet transactions that are person-to-machine. When machines talk to machines, their interactions trigger events, advertising something has happened. Some events communicate a result or trigger further events. Because of that, there is a need for speed of response. To compare request-driven and event-driven let us examine two interactions:

• A pedestrian comes to a corner where the light is red forbidding them to cross. They press a button on a nearby lamppost that says, “Press Here to Cross.” That pedestrian will wait for a considerable time for that light to change. This request-driven event anticipates some latency.

• An autonomous vehicle approaches the same corner intending to turn left. As it gets close, the light turns red, and cars coming the opposite way begin to proceed. If that event, the light turning red, experiences any latency whatsoever it may become a matter of life-and-death for several people. This requires an event-driven response that is consistently high-speed with no latency.

CHARACTERISTICS OF EDA APPLICATIONS

EDA facilitates high volume and high velocity of data such as traffic between things on the internet. More complex events may require identification of multiple events in diverse locations at different points in time, perhaps even on different systems, or multiple subsystems may need to process responses to the same event. EDA serves these scenarios bringing true real-time processing with minimum or no latency whatsoever in a completely decentralized platform that can even include systems from different organizations, critical in an interoperable IoT.

DECIDING WHEN TO USE AN EVENT-DRIVENARCHITECTURE

The decision to use EDA should consider the presence or absence of several characteristics:

Asynchronous and Loosely Coupled

Each component of an EDA application operates independently which allows them to move on to their next task even while downstream components are performing further processing. In fact, each service in the EDA has no knowledge or awareness of other components including protocols and other details. This also allows an event router to queue events received where they can wait to be processed when their intended event consumer becomes available. They can also be tested, validated, updated, and deployed independently.

Since the event queue in the event router stores every event, events may be “replayed” later for purposes of analysis and recovery of lost data.


Scaling

Absent any awareness of each other, the loose coupling of the various components of service makes scaling simple. Each service may be scaled independently without dependencies on any other components.

EDA significantly increases process velocity and agility, making it ideal for modern applications that use microservices, or any application featuring decoupled components. This requires shifting your paradigm to align more closely with EDA. Things to consider include:

• The resilience of event sources. Especially if the expectation is that you will service every event your event source must guarantee delivery.

• How you will troubleshoot anomalies. Since there is no designed workflow in an EDA system, it is not possible to trace errors through the code. You will want to create a strategy that monitors and evaluates each segment independently.

• Data management. Where earlier systems collected data and then processed and analyzed it before producing an outcome, with EDA the event itself is the data. If you ever need to rebuild the current state during any past event transaction, you will need to provide indexing and deduplication.

When decision time is measure in milliseconds instead of minutes, for example determining whether afinancial exchange is fraudulent while the transaction is in mid-flight, your best choice is EDA. Other examples include payment processing, website monitoring, online customer experiences, and other real-time marketing activities.

IMPROVING AI EXPERIENCES REQUIRES EVENT-DRIVEN ARCHITECTURE

As the use of natural-language AI-driven human interfaces increases, EDA will be critical to the naturalcadence of conversation required. If a user asks a question and the AI is delayed in replying, the illusion of true interactivity may be shattered. Latency is once again the enemy.

Event-Driven Architectures will also be pivotal in enabling the AI to consume events that may be relevant to its human user. An early example is the traffic function of some GPS devices. When heavy traffic or anaccident is detected ahead the AI is cued to advise the user and provide them with alternate routes.

EDA puts the digital side of the human/digital interface on more of an even footing, a level playing field with digital intelligence. When both can detect an event and decide to take specific actions in response to the concept of a truly symbiotic relationship between people and processors comes much closer to becoming a reality.

EDA ANDMICROSERVICES

Chapter 7:



The advantages of EDA covered earlier are mainly driven by the loosely-coupled, highly resilient way in which components interact. This aligns perfectly with the evolution of application creation.

The major presence of cloud computing finds us living in a new age of software development, a cloud-native application age.

Applications built for cloud delivery must be highly transportable, very loosely coupled, highly resilient, and extremely responsive.That’s a lot to ask for. Comparing today’s development environment to what came before will help to explain how all of this has been accomplished.

IN THE BEGINNING, THERE WAS THE MONOLITH

Anyone who has been developing for more than a few years remembers how applications used to bedeveloped, and still are in some corners, most of a given application was written as a single block of code. Every function, every Boolean option, every repetitive or iterative process, every service the application called for was contained within that code.

The flow of the code began at the beginning and proceeded on down, executing each command within each service in sequence until a decision-point was encountered. Perhaps a specific variable needed to be tested to determine where to proceed next.


Perhaps a user needed to enter a selection or response before processing could continue. Communication between each of the services, processes, functions, subroutines, and libraries was inherent in the processing of the code.

A neatly organized, complete package. Not only was that an advantage, but it was also a critical disadvan-tage. In a complete, monolithic application like this were anything to go wrong anywhere within the code, the entire application would completely come down. Program error. Critical error. Fatal error. Not pleasant, and often not easily resolved.

SOLUTION! JUST ADD GRANULARITY WITH A SIDE OFINDEPENDENCE

If a flaw occurring in any service could bring down the entire application, the logical solution would be to isolate each service by running it separately and independently. A failure in any service would only bring that process down, not the entire application, which would keep running until the failed service was re-instantiated and became available.

This thinking, which actually began decades ago, led to the development of microservices, small services that interact with other services to form and run an application. To increase the isolation of each service, it would run in its own process in a container that included the code for the service, its configuration, all dependencies, libraries, and other resources required to run the code. Containerized services can be individually tested and are deployed as a containerized image instance to the host OS.

There are several significant advantages to creating applications as an assembly of independentcontainerized microservices.

• Since they are each executed independently each can contain different code with differing dependencies created on diverse platforms.

• They can be deployed across varying environments with no modification.

• Running directly on the OS, containers have a much smaller footprint than VM images.

• Scaling out is easily achieved by creating new containers for various tasks. Instantiation of a new image, the process for creating containers, is not unlike instantiating a service or web app.

• Containers offer independence and isolation, portability, scalability, and control.

By interconnecting containers in a service mesh, you build cloud-native apps that run reliably across whatever environments they may encounter.


THE NEXUS

There is a nexus where all the latest innovations in software development meet. Microservices, containers, DevOps, continuous improvement through continuous development and deployment (CI/CD), event-driven architecture (EDA) and more all coalesce around the achievement of increased agility.

Developers enjoy a division of labor, forming small teams to build and maintain specific services. They can even build those services in any language since each service will run separately from all others.

Bringing this all together, containerized microservices align with the core concepts of agility. They are very loosely coupled, so a change to one does not necessitate changes to others. Since they are easily reproduced they are highly scalable. Should a change be required, only the service requiring the change needs to bemodified. And containers are literally the definition of granularity.

There is only one more piece required to bring them all together.

RESPONSIVE, INTERACTIVE, EVENT-DRIVENCOMMUNICATION

In the monolithic architecture of the past, everything happened within the over-arching application.Microservices each run independently from each other. So how are they going to communicate with each other? Certainly not in the classic way of waiting for action from a user.

Classic code was command driven. A command was issued by a user and the system ran the applicationcontaining all the required services. To eliminate the need for human intervention, the software would need to be able to detect that something has happened, an event, and respond to that event appropriately.

This was the driving force behind the development of EDA. Make it possible for a significant change in the condition of a component of the system to be recognized by the system, thereby triggering further action or actions by the system. Now microservices can run, produce a resulting event which is then handled by an event producer who processes it, sends it to the event router which ultimately distributes it among one or many event consumers responsible for further action.

To operate, containerized microservices require the kind of responsive communication provided by EDA.

AN EXAMPLE OF AN EVENT-DRIVEN MICROSERVICESARCHITECTURE:


Missing ringSensor triggered Theft processor

Notification

Sensor

Ring robbed event

Back doorSensor triggered

event

Activate alarm event

Call police event

Call owner event

Notificationrequired event

Thief stealsexpensive ring

Alarm Police Owner

A simple event often requires complex responses. In this illustration a premises sensor has detected the event of an expensive ring being stolen. Event processors provide the guidance required to provide deterrence by sounding an alarm while also notifying the police to respond and also notify the owner of the event. None of these notifications need be aware of the others, nor wait for them to occur before executing. The immediate action this sequence provides demonstrates the value of loose coupling.

CLOUD-NATIVE APPLICATIONS

Interconnecting containerized microservices creates cloud-native apps that can easily transport to wherever on the network they are needed. To run reliably and consistently they must have a communications platform that automates all potential responses. While classic monolithic applications could not scale as well, orprovide the resilience required, cloud-native apps take advantage of EDA to enable them to facilitate the agility that defines the goals of DevOps which is to achieve continuous improvement in a dynamic environment in which continuous development and deployment are highly facilitated.

Every Silver Lining May Have a Dark Cloud…

You may by now be worrying that this loosely-coupled, resilient world of continuous improvement sounds too good. There must be a downside.

Next, we’ll explore the disadvantages of using EDA. You’ll see this is not a one-size-fits-all architecture, but the good clearly outweighs the bad.


DISADVANTAGESOF EVENT DRIVEN

ARCHITECTURE

Chapter 8:



The real world is a messy state of affairs. Things happen for all kinds of reasons, sometimes for no apparent reason at all.

Other things and people react even though they may bear no direct connection with the thing that happened. For example, when a traffic light turns red it really doesn’t know whether drivers seeing it will stop their cars. And in a very real sense, it doesn’t care. Even if failure to stop causes a major accident the traffic light will just keep on doing its job unless one of the vehicles impacts the control box.

This puts the real world at odds with classic software development in which everything is regimented and proscribed by specific rules. The traffic light, for example, changes at programmed intervals without fail unless it becomes damaged. Classic monolithic design with all decisions determined and executed without regard to anything else happening outside the system.

The cars and their drivers, however, are event-driven. They will continue proceeding until they encounter something from the traffic control system, which is only connected to them by inference. If a light turns red they’re supposed to stop, but there is no programming that enforces or requires that to happen. One event is not connected or coupled to the other in anything other than a loosely enforced common understanding of the rules.

THE TOP 5 DISADVANTAGES OF EVENT-DRIVENARCHITECTURE

THE DOUBLE-EDGEOF LOOSELYCOUPLED EVENTS

DOCUMENTINGANTICIPATION OFTHE UNKNOWN

EVENT-DRIVENARCHITECTURE ISNOT A PANACEA

ERROR HANDLINGHAMPERED

ANTICIPATINGTHE UNFORESEEN

DISADVANTAGESOF EVENT-DRIVEN

ARCHITECTURES


1. The Double-Edge of Loosely Coupled Events

One of the more obvious ways in which the real world intersects with programming can be found on theInternet of Things (IoT). Various sensors recognize actions taken by humans and set off responses to them. The sensors are tightly-coupled to the systems that perform the responses, but can only be consideredloosely coupled to the person setting them off.

Extending this, the response taken by one system may trigger events in other systems. Each system is very loosely coupled to the others and cannot anticipate any particular event happening at any particular time. By employing event-driven architecture (EDA) we enable all systems to react in specific ways to events triggered by other systems.

Given the vast number of “things” connected to the IoT it would be impossible to anticipate all possible events and tightly couple them to all others. Having systems able to detect and react to events that take place in time enables the interoperability absolutely necessary to the functioning of the IoT.

At the same time, EDA operates within a void of connectivity. Since a system that produces an event is not directly coupled to systems that react to it, space is created between them which becomes a point ofvulnerability that could be exploited by those of bad intent.

Loose coupling is required to achieve scale but must be evaluated carefully to protect or prevent security gaps.

2. Documenting Anticipation of the Unknown

Developers live in a world they themselves proscribe. Much time and effort go into anticipating all the various possible outcomes from each step in their program and making allowances for them.

This is the antithesis of EDA in which there can be little or no anticipating which systems will produce what events at any given time. While the developer may react to this reality by attempting to program theproduction of as many events as possible to achieve greater granularity, the reality is that this may easilybecome counter-productive as it adds greater complexity which only reduces the ability to trace thelogic, test, and troubleshoot while also potentially overwhelming event consumers with an overabundance of events to process.

Due to the inherent complexity of creating software that evaluates events rather than objects and must allow for unrecognized events, malformed events, or other anomalies, the EDA learning curve can be extensive. This is exacerbated by how opposed EDA thinking is to classic software development architectures. Replac-ing a world in which every possibility was contained within the system, known and provided for, now much of what the system must process occurs outside the system. As such it cannot be documented.


Within software designed with EDA, developers must take care to avoid generic names and flags for events their software produces so they and others may evaluate and modify the code in the future if need be.

3. Anticipating the Unforeseen

Testing is difficult in an EDA environment, which is designed to anticipate the unknown but also may trigger unforeseen and undesirable responses. In classic waterfall, programming testing includes seeking potential loops with the potential to become endless. Endless loops are every bit as possible between disparatesystems that react to each others’ events as they are within monolithic code, just far more difficult to test for where there are far too many potential permutations of action.

Since events are asynchronous there can be no anticipation of a specific order of occurrence or assured deliv-ery of anything. Duplicates may occur in differing conditions that may each require a contextual response.

4. Error Handling Hampered

Perhaps the most confounding challenge in using EDA is the difficulty in providing root cause analysis of any given failure. As event producers and event consumers proliferate it becomes more and more difficult to trace back activities occurring within and between them. Like an interstate automobile pursuit, the most difficult transition occurs when the getaway vehicle reaches and crosses over the border where the chase must then transition between the first state’s law enforcement vehicles and the second. This is an ideal place for the getaway car to getaway.

5. Event-Driven Architecture is Not a Panacea

It is very important that EDA developers not come to view EDA as a solution for every problem, a cure-all. Given that it circumscribes each process it is preferable to design but obtaining that value may not be worth the complexity introduced by all the interprocess interaction. EDA enables the interactivity and interoper-ability that the growing IoT runs on, but that doesn’t make it the right answer for everything. It is critical to evaluate the interactivity and interoperability required by any given use case to determine if EDA is really called for. It may require too much troubleshooting and maintenance to be used in situations that simply don’t need it.

In the many cases in which EDA is just right, it becomes important to understand how to overcome these disadvantages.


OVERCOMING CHALLENGES OF EVENT DRIVEN

ARCHITECTURES

Chapter 9:



Event-Driven Architecture (EDA) solves many of the difficulties encountered when building and operating classic monolithic applications in which a failure of any service will bring the entire application down. Instead of responding to commands or requests. Services designed using EDA are very loosely coupled to other services. Each awaits the occurrence of specific events to trigger them to take specific actions. Failure of any service will be quickly resolved by immediate re-instantiation of that service. The entire application is never threatened since it is fundamentally an assembly of many individual services.

We’ve now discussed the many advantages of EDA and reviewed the challenges that often face thoseattempting to convert existing monolithic applications into microservice-based event-driven systems or build new systems using EDA. Now let’s look at how these challenges are overcome.

ADAPTATION TO A DIFFERENT APPROACH

Solving for many of these challenges requires developers to more aggressively abandon their existing para-digms and preconceptions. Some of what they learned and practiced in building SOA-based applications may apply in an EDA environment, but many run counter to it. They may have difficulty in adapting to the more granular environment of EDA. They may remain more comfortable with their representational state transfer (REST) approach. Their early efforts may be marked by chatty services and continuing to insist that databases are a shared resource.

There is also a natural tendency to avoid the risks created by such radical change as the change frommonolithic to microservices and EDA.

IMPROVING AND ADDING NEW SKILLS TO OVERCOME EDA CHALLENGES

Tracing code in a monolithic application is straightforward since all the code for all operations is contained within the monolith. Developers will need to learn the principles of tracing events that occur betweenservices and the reactions those services will produce. A tremendous change of mindset is required to make this transition.

This change of mindset will create a need for the adoption of new tools. Prior tools were not designed to take advantage of new developments such as microservices, containerization, polyglot environments, and more.

In an EDA environment, the greatest development challenge will be identifyingdependencies and interactions between various services.

Rooting out problems will be impossible without using tools designed to assist.


GOVERNANCE IS CRITICAL

Especially given the enhanced ability to have multiple teams developing different services that will allultimately need to interoperate, the establishment of standards and a process for assuring adherence will be critical to improving time and cost of usable product. Nomenclature, variable labels, and others must becarefully aligned between domain owners. Not only will code development benefit from standardization, so do the various databases each service interacts with. Even logging of policies, procedures, and frequency should be aligned within governance.

Also, protocols and standards must be shared across all teams to assure that the application will perform flawlessly.

Aligning tools used between teams is also valuable to overcoming many challenges. Team training and orien-tation should be provided regarding tool selection and application. Continuous monitoring of these processes will be accomplished through consistent evaluation of logs.

Team governance is also required to assure that each team is not only composed of developers and technol-ogists, but also of domain experts whose purpose is to keep the services aligned with the business operation objectives of the project. Returning to the mindset issue, EDA development cannot be technology-biased or data-centric. EDA is best employed to respond to business-relevant objectives. One key manifestation here is to focus on database transactions, but EDA should always be focused on business processes.

REVISITING THE MAJOR CHALLENGES OF EDA

Loosely-coupled events, services, and processes are key components of any EDA strategy. Services and events only become more valuable when the focus is maintained on the business processes they support, and not the databases.

While it may represent something of a leap of faith, business and workflow logic will sooner facilitatedevelopment and tracking of EDA code and process flow. Since the logic is most often altered by response to events occurring between services it becomes difficult to follow the logic technologically. It is far simpler to follow the intended workflow in terms of business operations to assure that each event triggers appropriate responses.

The unknown results that can occur between different microservices is most often analyzed by developers with deep technical knowledge. But for the domain expert, what should happen in the normal flow of business is much more apparent, not really unknown. Pairing their domain expertise of what should happen with the developer who is building the code, is a powerful way of making the unknown known where it counts most, in the services themselves.


The unforeseen becomes more and more challenging as the ecosystem scales, and is harder to providecontingencies for as undesired results are often not occurring within any one given microservice. Deepunderstanding of the business processes supported is required to anticipate the unforeseen and provide a more resilient application ready to handle a greater scope of possibilities. One area of particular sensitivity relates to the sequence in which things occur, which may be highly variable. The best response to eachpossibility in each possible sequence.

When EDA is NOT the Best Solution developers must be able to see and recognize that EDA is not the best solution for any given application. With the proper granularity, the right combination of domain and technical expertise, and the careful identification of all possible events and their responses, EDA enables microservices to become the foundation of a highly interactive, interoperable solution.

So, finally, how do we begin?


EVOLVINGYOUR SYSTEM BYIMPLEMENTING EVENT-DRIVEN

Chapter 10:



Real-time is no longer a “nice-to-have.” Sensors deployed in the Internet of Things (IoT) are detecting events happening that require rapid response either from another machine, or sometimes from a human. Theexpectation that everything will interoperate with everything is no longer an assumption, its a requirement. You can implement event-driven architecture (EDA) to achieve it, but that imposes the challenge of changing the way you think about systems.

EDA enables real-time notification and response for every event happening within a system, or evenwithin associated systems. Instead of waiting for a command or request to be issued, EDA is constantlyvigilant awaiting for any of a large number of events to occur. When they do, notificationis sent to an eventprocessor that takes appropriate action in response to the event. With an event router, multiple responses may be triggered. Also, multiple events happening at different times in different places may be combined to trigger response once they all achieve their necessary states.The difference between a command and atriggering event is simple to visualize. You walk into a room equipped with a motion-detector light switch. When you do, the event of your entry is detected, the control for the light is notified and it goes on. That’s an event. Alternately, had you had to flip a switch to turn the light on, that’s a command.

EVOLVING TO AN ASYNCHRONOUS, REACTIVE MODEL

The word reactive carries something of a negative connotation. Everyone wants to be proactive, right? But in the case of the evolution of application development that’s exactly where we want to go.

Traditionally, developers were trained to think in procedures. Remote procedure calls (RPC), synchronous function calls, web services, APIs and service-oriented architecture (SOA) were the underpinnings of how applications were designed and built. All were synchronous, issuing a command complete with necessary data and metadata. All functions were tightly coupled to each other, so modification was a substantial undertaking. Each had awareness of each other, each command expected a response. The fact that all code could easily be traced in the event of a malfunction was helpful, but synchronous systems were difficult to scale.

EDA is loosely coupled and asynchronous. Events may be sent directly to an event broker or mediatordepending upon the complexity of the response required. Where a command requires a response, an event may or may not trigger a response, or the event processor may need to wait for all involved events to occur before taking action. None of the components of the architecture are aware of each other. Instead of sending a message and awaiting a response, all they do is send. The separation of every function makes it far easier to upgrade and improve components of the system without having to take the entire thing down.

GETTING STARTED WITH IMPLEMENTATION OF EDA

Anyone implementing DevOps quickly learns that the most important task is to develop a DevOps culture within the organization. Anyone participating will need to change the way they approach the design andarchitecture of solutions. They must trade in the long-conditioned tendency to see everything as a sequence of request, process, reply, confirm.


There will be no more program calls, requests, declarations, or invocations. There will be events that require processing, and responses that will often need to be produced.

As you shift paradigms from request-driven to event-driven you’ll also shift your approach from one oforchestration to one more closely resembling choreography.

The difference is simple. Orchestration tightly controls everything, all components, all interactions, remaining involved in each until it is completed. As you can imagine, that won’t scale readily as the orchestrator must await a response from each service it initiates before proceeding. Choreography provides patterns androutines that services follow, and it is expected they will do so without supervision or further instruction.

Tracking the flow of an EDA system is almost impossible since there is no set sequence of events. What happens in the system depends upon events taking place elsewhere in the system, or externally. The same is true for planning an EDA solution. Anything that can occur within your organization must be seen as a digital event. Every operating process is a series of events. There are no recursive loops for confirming activities.

Planning for Implementation

The planning process concerns itself with identifying the events that may happen at any given point andmapping the required responses to each under whatever combination of conditions they may occur.

Development

EDA will require that you decompose your existing workflows into microservices and implement runtimefabric that supports the ability of these to communicate in EDA’s preferred publish/subscribe (pub/sub)fashion capable of distributing to many potential destinations.

Required runtime and design components include:

• Event Broker: The most basic component for event routing in a pub/sub, providing low latency and guar-anteed delivery. Decoupled applications and microservices converse through the event broker preferably using open protocols and APIs.

• Event Mesh: A network of event brokers for routing events between applications on premises, in the cloud, or at the network edge.

• Event Portal: Similar to an API portal provides a view into your event mesh, giving architects design tools that are GUI-based. Defined events, microservices and applications can also be designed in the event portal.

• Event Taxonomy: The topic naming convention you design first and use consistently later. A good taxono-my supports event routing. Best to keep the conventions obvious and easily recognizable for developers.


AN EDA PILOT TO PLAY IT SMART

This is a dramatic change for everyone involved, a change in the fundamental way they think about andapproach development, deployment, and operation. Success will depend upon winning buy-in from every group that becomes involved, until the entire organization is familiar and comfortable.

For your pilot EDA project, choose something of moderate criticality from which the improvements will be readily observed and experienced. Moving from the comfortably confirmed communications world of syn-chronous systems to the highly decoupled environment of EDA can feel like a “crossing the chasm” moment. It is strongly suggested that you seek out expert, experienced assistance as you embark on your journey.


LEVERAGING APARTNER FOR EDA

SUCCESS

Chapter 11:



From reading this eBook you’ve hopefully learned a great deal about Event-Driven Architecture (EDA) and its place in the new, interoperable, machine-to-machine (M2M), Internet-of-Things (IoT) world. You’ve learned how it replaces commands issued with recognition of significant events occurring and automates response to those events in many cases performed by other systems. We’ve explored the advantages and challenges encountered when using EDA. If anything has become clear, EDA’s time has arrived and will be a welcomearchitecture for current and future development.You may even have some projects on the stack right now that would be better developed in the EDA architecture! The question then becomes how to minimizetime-to-value.

ACHIEVING THE LONGEST TIME-TO-VALUE

Certainly not something you want to do but worth discussing up front. Eventually, you’ll want to have all your developers and architects learn everything about EDA, but should you choose to do so in preparation for your first EDA project they’ll miss out on excellent opportunities to gain insights that will help them derive a far superior experience from whatever training you or they choose to take. Even more critical, you’ll be adding the duration of the training plus their needed trial-and-error practice time putting new skills to work to your project. Your time-to-value will be considerably longer which serves no one.

THE TWO ESSENTIAL ELEMENTS – EXPERTISEAND EXPERIENCE

You have several strategies available to you to bring needed expertise and experience to your EDA project, including:

Hire Someone

Preferably, you’d like to bring someone to your first EDA project who has experience using the architecture. Seeking to hire such an expert is full of significant challenges. Given your own unfamiliarity, how will you evaluate your candidates? Where will you find candidates? How much compensation will a capable candidate want? What if you choose the wrong person? Simply put, the hiring route may incur delay similar to training your existing staff once again delaying time-to-value.

Contract Someone

You’re familiar with the Gig Economy and know of individuals who contract themselves out regularly to firms like yours. Perhaps you can find one who is already up to speed on EDA. The chance is there that they will be available, reasonably priced, and manage to make it through the entire project without a hitch. Or they may fall ill, find another gig they like better, or make horrendous mistakes that stop the project. You really don’t know going in what to expect. You can only hope for the best, and hope is not a strategy. Fortunately, if they default on their agreement with you its fairly simple to put an end to it and start over. Not something you’d like to do.


Engage an Expert Firm

Today, partnering is the key to so many challenges. Finding a suitable firm is far easier than locating aproperly-skilled person. A quality firm will field many skilled experts in EDA as well as many other valuable and likely adjacent skills and capabilities. EDA is not new, but it is only now picking up momentum and aprofessional firm will depend upon being prepared to help clients before anyone else.

When selecting a firm to engage, certainly inspect what you expect in the way of expertise. You might want to look for a partner with well-known industry certifications from big tech and cloud providers. Since EDA can benefit from a whole range of skills from software architecture, DevOps, Microservices and other technology integrations, those certifications can be a signal that you are talking to a more experienced firm.

Next, go beyond their expertise. One thing that may often be in short supply is experience. Not every firm has encountered EDA projects as yet. Others simply haven’t adopted EDA in their practice. You don’t want to be their test case.

Before you even begin to interview a firm ask to review their standard statements of work, master service agreements, and other legal documentation. EDA projects are simply not trivial nor are they inexpensive. Nobody wants to have to abandon aproject midstream to change providers, nor do you want to discover you cannot recover fees for which you have not received value.

Perhaps even more important are the protections specified in the agreement. You’ll likely be exposing a great deal of information to your selected firm. Your fiduciary responsibility requires that you assure protection of your company’s proprietary and intellectual property. Since many of your own employees will be interact-ing with members of the chosen firm you also want to assure protection against them hiring your people or otherwise interfering with your business. You’ll also want to confirm ownership of any code or other products of this engagement to assure that you won’tend up “locked in” to them in the future, nor find yourself paying repeating royalties.

Working Together

When talking about any technology, many people forget that the working relationship they have with those they partner with is paramount. The success of your EDA project may easily depend more upon the“chemistry” between you and your chosen firm than any other element.

Sit down and meet with them, either virtually or in person, before any discussion of the project commences. Get a feeling for who they are, what they value, what their standards are, and how they prefer to work with clients. Focus on finding out what their approach to knowledge transfer is. You definitely want to take fulladvantage of the value of having your own people learn from their people. This will not only serve during operation of your system, it will also benefit them in the value they’ll enjoy from future training.

If possible, also explore their reputation by searching for them on social networks and search engines. Your diligence up front will pay back repeatedly as your project progresses.


About Tiempo

Tiempo Development’s IoT consulting services are designed to address our clients’ unique challenges and help them reach key objectives--whether that means closing the skills gap, securing your smart system, or implementing the cultural changes that pave the way for transformation.

Tiempo is widely recognized as one of the leading software engineering companies in the US. Using a combination of nearshore engineering resources, high-performance teams and relentless focus on client outcomes, Tiempo designs, builds and deploys software that makes lives better.

Tiempo is headquartered in Tempe, Arizona, with four worldclass software development facilities in Mexico. Tiempo has been recognized annually by Inc. Magazine as one of the Fastest-Growing Private Companies in America.

U.S. HEADQUARTERS1050 W. Washington St, Suite 115 Tempe, AZ 85281 USA

GUADALAJARA2464 Justo Sierra Guadalajara, Jalisco, México

GUADALAJARASan Dionisio 25 Jardines de San Ignacio Zapopan, Jalisco, México

HERMOSILLO545 García Morales Blvd. Hermosillo, Sonora, México

MONTERREYValparaiso 49 Col. Alta Vista Monterrey, Nuevo León, México

www.tiempodev.com

Contact us at:[email protected]

https://twitter.com/tiemposoftware?lang=en

https://www.facebook.com/TiempoDevelopmentLLC/

https://www.facebook.com/TiempoDevelopmentLLC/

https://www.linkedin.com/company/tiempo-development/

https://www.linkedin.com/company/tiempo-development/

https://www.youtube.com/channel/UCstWL43CIcNUd7myKeSXeVg

a business leaders guide to event-driven architecture · event-driven architecture this ebook...

Documents