building an esb with websphere mq and message broker

© 2006 IBM Corporation

GSE Nordic WebSphere User Group 2006

®

Building an ESB with WebSphere MQ and Message Broker

Jonathan [email protected]

Message Broker Development

IBM Software Group

2Building an Advanced ESB for SOA with WebSphere Message Broker Version 6

Agenda

Service Oriented Architectures• What is a service and why do I need SOA?

Enterprise Service Bus• What is it and what does it mean for SOA?• Functions of an ESB

Protocols, Route, Transform, Eventing• What is an Advanced ESB?

Building an ESB with WebSphere MQ and Message Broker• Features and functions to build an ESB

Summary & References

IBM Software Group


Service Oriented Architectures

IBM Software Group


What is a Service?

(Ideally) a self-contained, stateless business function that accepts one or more requests and returns one or more responses through a well-defined, standard interface. Services perform

discrete units of work such as editing and processing a transaction. Services should not

depend on the state of other functions or processes. The technology used to provide a

service, such as a programming language, does not form part of its definition.

IBM Software Group


Applications implement most of today’s Services• There’s nothing wrong with this!

How are services and applications related?

• Application can implement the Service exactly

• Service can be implemented within part of an Application

− Including multiple services

• Combination of Applications can implement the service

Services and Applications

Application ServiceKEY:

IBM Software Group


Identifying Services

Technology reality•“Where I am right now”•“Bottom Up” approach•Example: CICS transactions, MQ applications, Files, JSP, EJB…

Business driven•“Top Down”•“Where I’d like to be in the future”

−Explicitly user and business focussed

Mixed Business and Technology•“Middle Out”•Emphasis on cooperative and evolutionary approach

IBM Software Group


What is a SOA?

A software architecture which

• …defines the use of services to support the requirements of software users. Resources are available to other participants as independent services accessed in a standardized way.

• … comprises loosely coupled, high interoperable application services. These services interoperate based on a formal definition independent of the underlying platform and programming language. The interface definition hides the vendor and language-specific implementation.

• … is independent of development technology (e.g. Java, .NET, COBOL, C, ASM). The software components become very reusable because the interface is defined in a standards-compliant manner.

IBM Software Group


How do you know when you’ve got SOA?

+ = + =

Easy to connect services together without knowing that they had to be connected initially

• Take components / services and add them together• Then, later add more services / components

IBM Software Group


What are the benefits of SOA?

Asset reuse• Service is component of reuse

Lower Development Cost • Eliminate duplicate services

Faster Time-to-Value• Compose new applications from existing services

Extend Business Reach• Allow new clients to exploit existing services without change

Providing a new front end

Business Responsiveness• Dynamically reconfigure services to meet new business opportunities

Governance• Standardize, mandate and measure service usage

Allows you to introduce control

IBM Software Group


Enterprise Service Bus

IBM Software Group


What is an ESB?

A software infrastructure for SOA which

• … comprises a set of interaction points to which services are connected. Services are able to invoke other services which are connected to the bus.

• ... simplifies an SOA by minimizing the explicit connectivity between services.

• … allows the creation of dynamic and flexible connectivity between services during service invocation without changing service consumers or providers.

IBM Software Group


SOA with an ESB

Simplification of Infrastructure• It DOES simplify the way you think

about service connectivity. • It DOESN’T change the way you think

about services.

Dynamic and Flexible Infrastructure• New connectivity

e.g. add customer request audit.• Flexible connectivity

e.g. prioritize customer request. • Service replacement

e.g. New service upgrade without…

IBM Software Group


Functions of an ESB

Four functions in an ESB• Protocol support• Routing• Transformation• Eventing, including CEP

Other functions…• Security

can be performed in ESB, but not core functionality

Shape = ProtocolColor = Data type

IBM Software Group


What are the different kinds of ESB?

Basic ESB• XML as only supported data format• Web Services protocol emphasis (SOAP/HTTP)• Adapters for everything

XML conversion on and off the busProtocol conversion on and off the bus

Advanced ESB• All formats supported natively

XML, COBOL, C, SWIFT, EDI, MIME, Acord, X12• All protocols supported natively

Web Services PLUS MQ, JMS, File, SCADA device, user defined…• Minimizes need for adapters

Improves performance and manageability

IBM Software Group


Building an Advanced ESB with MQ and MB

IBM Software Group


The Advanced ESB

WebSphere MQ and Message Broker form an Advanced ESB

MQ is transport protocol of choice for high performance, transactional messaging

• Very high quality of service transport• Loosely coupled

Message Broker provides higher level ESB functions• Comprehensive range of native transport protocols

FROM anywhere, TO anywhere• Perform suite of routing actions on service messages• Transformation of message formats between services• Broad range of Event Processing capabilities

IBM Software Group


Use MQ to Build SOA Enabled Applications

WebSphere MQ

• An excellent SOA interface for applicationsSimple and powerful interaction styleVery high quality of service transport

• View your applications as servicesSelf contained, well defined, standard interfaceLoosely coupled requests and responsesDiscrete, transactional units of work

MQ

Application ServiceKEY:

IBM Software Group


Add Message Broker for an Advanced ESB

WebSphere Message Broker

1… creates services from applications.

2 … provides service interactions for a broad range of protocol endpoints, including MQ.

3 … provides flexible and dynamic connectivity between service endpoints.

Provide routing and transformation capability

4 … combines with other brokers to form a scaleable ESB backbone.

1

24

3

Application Service

KEY:

Input & Output nodes

Message Flow

Protocol

IBM Software Group


ESB Function 1 – Protocol Support

Consume and generate service request messages• “FROM anywhere, TO anywhere”

Ability to bridge between application services which use different transport protocols

Extensive range of transports means only need to use protocol adapters if necessary

MQ is transport protocol of choice for high performance or transactional interactions

IBM Software Group


Message Broker provides a comprehensive range of protocol endpoints using input and output nodes.

Also including…

Protocols - Message Broker Input and Output Nodes

UUser DefinedMQ HTTP SCADA

JMS 1.1 MQe IP/Multicast

Database

FLAT FILES VSAM/QSAM

SMTP

CICS TCP/IPTIBCO RV

IBM Software Group


Protocol Support

Grouped into different categories of protocols

de facto Standard protocols• MQ, HTTP, JMS, TCP/IP

Native input nodes

“Standards” protocols• Web Services via SOAP/HTTP, SOAP/JMS

Native HTTP(S), Any JMS 1.1 connectivitySOAP and MIME parsers provided

Technology interactions• Files, VSAM, QSAM & CICS to access traditional data sources

Industry protocols• SCADA, MQe for mobile devices and sensors

User Defined protocols• For User ISV provided extensions, full user defined node interface.

IBM Software Group


ESB Function 2 – Routing

Routing can take on different forms• Route, Filter and Order service request messages

Dynamically direct service requests within the enterprise

Support for many types of routing within different interaction patterns

• One-way• Request-response• Publications• Service decomposition and reassembly

IBM Software Group


Message Broker provides a powerful suite of routing nodes.

Flexible and wide range of routing metaphors supported.

Routing – Message Broker routing nodes

Route and Filter

Order

AggregationPublication Trace

Exceptions User Logic

IBM Software Group


Message Broker provides an Eclipse based Message Flow Editor.

User friendly tool to express routing connectivity.• Drag and drop nodes onto the canvas• Input and output nodes represent the protocols• Lines connecting nodes show the ‘route’ through the flow

Routing – Building a routing flow

IBM Software Group


Routing

Full range of messages can be routed and filtered• XML – for XML payloads (full schema support)• Record based – for COBOL, C payloads• Industry standard – SWIFT, EDI…• User defined – RYO formats

Different routing and filtering rules• Message based

Message Header and Body routing are supported• External reference

Database lookup, PLUS in-memory cache for infrequently changing data• User algorithm

Partial Parsing for efficient routing of large messages

IBM Software Group


ESB Function 3 - Transformation

Connected services may require different message formats• Shape• Data format

Ability to transform BETWEEN service message formats requires• Model of message formats• Transformation language to map between formats

Wide range of Message formats supported natively• No need for extra XML normalization• Model created using importers (COBOL, C, DTD, Schema) or Eclipse tooling

Wide range of Transformation languages• Allowing you to choose the right tool for the job• Matches existing or future skill sets

IBM Software Group


Transformation – Graphical Mapping and Java

IBM Software Group


Transformation – ESQL and XSLT

IBM Software Group


Transformation

Message Broker models all data via “Message Tree”• All data formats (XML and non-XML) are understood natively by Message Broker• Message Tree created by a native parser from the source input message format• Transformed using appropriate transformation language• Output tree used to generate required target output message format• …leads to high performance

Data is only transformed when it is necessaryPartial parsing only reads required fields

Use transformation languages to match skill set• Graphical mapping for non-programmers• XSLT for XML orientation• SQL for database and declarative orientation• Java for widely available skill set benefits

• Transformation languages can be mixed and matched within same message flow

IBM Software Group


ESB Function 4 – Event Handling

Processing external events• Ability to distribute information to interested parties

Publish Subscribe and more complex event processing (CEP)

Timer based processing• Periodically start flows at preconfigured and programmatic time interval

IBM Software Group


Eventing – Message Broker event nodes

Message Broker provides a rich set of event nodes.Publish Subscribe Timer Complex Event Processing

IBM Software Group


Complex Event Processing - What is it?

Active Systems are systems that contain active (event-driven) components• Reactive Systems – React to something that happens in the system (server failed, direct

requests to other servers)• Proactive Systems – Use predictive methods to redirect the system towards better results

and or eliminate problems (server utilization is high, direct request to other servers)

Processing of action triggered not by a single event, but by a complex composition of events, happening at different times, and within different contexts

Examples: compliance checks, fraud detection, monitoring of service level agreements, etc

IBM Software Group


Complex Event Processing Nodes

SituationManager Node

Detected situations result in message being written to queueDetected Situations

IntelligentFilter Node

Detected situations result in alternate path through messageflow

IBM stock gained 3% and NASDAQ lost 1% within the last hour

Route messages to a compliance officer if a client performs more than a certain number of messages that cause a credit breach in one day

IBM Software Group


Security Considerations for an ESB

Can secure the link between the service and ESB• Transport protocol security, e.g. SSL

All security cannot be done in ESB• Co-operation from the protocol / transport• Extreme is that none of the security is

handled within the ESB

ESB needs to “see” relevant data to perform routing and transformation• Security versus flexibility

IBM Software Group


Performance Considerations for an ESB

Flexibility AND performance are metrics of success• ESB can affect latency, possibly throughput

Partial parsing for efficient routing and transformation of large messages• Only parse as much as necessary

50% average throughput improvement for Message Broker v6• Same flows, etc.

Important to use performance enhancing features where possible• In-memory caches• Queues, rather than databases, for transient data

IBM Software Group


Summary

ESB is an important infrastructure component for SOA• Protocols, Routing, Transformation, Eventing

WebSphere MQ and Message Broker form Advanced ESB• Integrate standard and non-standards-based applications and services

IBM Software Group


References

WebSphere MQ and Message Broker

http://www.ibm.com/software/integration/wmqhttp://www.ibm.com/software/integration/wbimessagebroker

IBM Enterprise Service Bus web site:

http://www.ibm.com/software/info1/websphere/index.jsp?tab=landings/esbbenefits

ESB Primer

http://submit.boulder.ibm.com/dd/wsdd/asis/501957/501957.html

Red BookPatterns: Implementing an SOA Using an Enterprise Service Bus: SG24-6346

IBM Software Group


Summary: Resources … IBM’s ESB Web Site

IBM Enterprise Service Bus web site:http://www.ibm.com/software/info1/websphere/index.jsp?tab=landings/esbbenefits

IBM Software Group


Any Questions?

IBM Software Group


Extra foils

IBM Software Group


MQ, MB and the SOA Lifecycle

IBM Software Group


SOA Lifecycle and ESB

SOA Lifecycle is model, compose, deploy, monitor

ESB is infrastructure component of SOA• Model service endpoints• Compose service endpoint connectivity• Deploy endpoints and connectivity to ESB• Monitor ESB

IBM Software Group


SOA Lifecycle – Model and Compose

Application

Service

KEY:

Input & Output nodes

Message Flow

Protocol

CICS TXN

MQHTTP EXCI

CONSUMER

MQ

CONSUMER

HTTP

IBM Software Group


SOA Lifecycle - Model

How to model a service endpoint• Service is an application, e.g. CICS, IMS transaction• ESB makes application endpoint available to other services

Formal Service definition• None (implicit)• Paper • COBOL/PLI copybooks, C header files• WSDL• SCA• …other

Tools for service endpoint definition and interactions• Importers to create ESB service endpoint definition from application• Exporters to enable ESB endpoint invocation from application

IBM Software Group


SOA Lifecycle - Compose

Two types of composition• Service creation

Composing services from the applications you already have• New service creation

How to (de) compose a service from other services• Both aspects are important

Work with what you haveKnow how to evolve

Comprehensive samples in Message Broker version 6• Most common scenarios available for import directly into work space

Request-reply, Web Service, Audit, Aggregation…

IBM Software Group


SOA Lifecycle - Deploy

Make an ESB service endpoint active• Message Flow is made live

Servicing requests, and generating responses• Hot deploy is possible

Existing endpoints are not affected• Deployment tools

Eclipse, Command Line, Programmatic (Java or PCF)

Runtime Versioning• Ability to determine which level of ESB artefact is active

Message Flows, Message Sets etc…• Trace-ability• Problem determination

IBM Software Group


SOA Lifecycle - Deploy

IBM Software Group


SOA Lifecycle - Monitor

Monitor ESB connectivity layer• Monitor Flows, Execution Groups, Brokers

Application monitoring is outside scope of ESB• Need integrated monitoring tools• This cannot be delivered by individual products

SOA based monitoring tools e.g. ITCAM for SOA

End to end tracking becoming increasingly important• Correlation between messages through Application and infrastructure• Message Broker V6 Message Flow tracking feature

Monitoring tools enabled• Tools to consolidate across enterprise wide interactions

IBM Software Group


SOA Lifecycle – Monitor ESB

IBM Software Group


Monitoring – the wider pictureAs SOA’s get deployed you need to see all of the “bits” using one view, you don’t have

time to hop from monitor to monitor when your business is effectively stopped.

You have to implement things such as ITIL service desk functions to restore normal operations as soon as possible – using tools such as ITM 6.1 and OMEGAMON Reflex Automation.

You will have to integrate components into your infrastructure that do not have traditional monitors – Tivoli’s UA Technology can build these and interface directly into the TEP (Tivoli Enterprise Portal).

As Z Series continues to grow it hosts the major components of SOA’s – DB2, CICS, WMQ, Broker, WAS, WPS, ESB

Business users will want to charge for Services – which means SLA’s and SLA compliance and Business Views – “How many widgets did we sell today ?” – not how is CICS response time.

IBM Software Group


Managing an ESB

Tivoli will exploit the brokering runtimes, mediation frameworks provided by WebSphere• Create and manage the mediations for management

Tivoli management server displays aggregated data, provides notifications, alerts for IT Operators

• .Net, WebLogic, WAS

ServicesIntermediary

Service Requestor

Service Provider

RoutingTransformation

Monitoring Enforcement

TEP Views provided by:ITCAM for SOAITCAM for RTTITCAM for WAS

OMEGAMON XE for WBI

= ESB functions provided by WebSphere

= Management functions provided by Tivoli

IBM Software Group


Service Operations View

Service Operations View

Service Inventory View

Service Inventory View

Service Performance View

Service Performance View

Integrated views of web services data through a centralized portal

Aggregate web services data with other IT infrastructure monitoring

Customize workspaces to tailor views to specific roles

Use Situations to create thresholds, alerts and take actions

Provide message logging as well as message rejection

Manage Service Relationships

IBM Software Group


Summary

IBM Software Group


Summary

ESB is important infrastructure component for SOA• Protocols, Routing, Transformation, Eventing

WebSphere MQ and Message Broker form Advanced ESB• Integrate standard and non-standards-based applications and services

IBM Software Group


References

building an esb with websphere mq and message broker

Documents

advanced esb

esb protocols

components services

services components

ibm software groupwhat

use of services

ibm software groupsoa

ibm software groupservices