introduction to the websphere message broker global cache

7/25/2019 Introduction to the WebSphere Message Broker Global Cache

1/12

4/14/2016 Introduction to the WebSpher e Message Broker global cache

http://www.ibm.com/developerworks/webspher e/library/techarticles/1212_hart/1212_hart.html

Share:

developerWorksPremium An all-access pass to building your next great app!

Sign up

Thenew global cache feature in WebSphere Message Broker V8.0.0.1 enables you to share an in-memory cache

between processes and brokers. This article describes the global cache, shows you how to use it, and answers some

frequently asked questions about it.

James Hart is a Software Developer on the WebSphere Message Broker Development team at the IBM Software Lab in Hursley Park, UnitedKingdom. He has worked there since 1999 in a variety of technical and customer-facing roles, and he is currently Technical Lead for cachingtechnology in WebSphere Message Broker.

19 December 2012

Also available in Chinese

IntroductionThis article provides an overview of the new global cache capability in IBM

WebSphere Message Broker V8.0.0.1. The global cache uses WebSphere

eXtreme Scale technology to provide an in-memory data cache that can be

shared across execution groups and brokers. This article has three sections:

Overview of global cache

Frequently asked questions

Cache topology considerations

Programming considerations will be covered in a future article.

Overview of global cache

Why was this capability needed in WebSphere Message Broker?

A longstanding WebSphere Message Broker requirement has been a mechanism for sharing data

between different processes. This requirement is most easily explained in the context of an asynchronousrequest/reply scenario. In this kind of scenario, a broker acts as intermediary between a number of client

applications and a back-end system. Each client application sends, via the broker, request messages that

contain correlating information to be included in any subsequent replies. The broker forwards the

messages on to the back-end system, and then processes the responses from that system. To complete

the round-trip, the broker has to insert the correlating information back into the replies and route them

back to the correct clients.

When the flows are contained within a single broker, there are a few options for storing the correlating

information, such as a database, or a store queue where an MQGet node is used to retrieve the

developerWorks Technical topics WebSphere Technical library

Introduction to the WebSphere Message Brokerglobal cache
https://www.ibm.com/marketplace/cloud/developer-program/us/en-us?cm_sp=dwpremium-_-1212_hart-_-adhttps://www.ibm.com/marketplace/cloud/developer-program/us/en-us?cm_sp=dwpremium-_-1212_hart-_-adhttp://www.ibm.com/developerworks/websphere/library/http://www.ibm.com/developerworks/websphere/http://www.ibm.com/developerworks/topics/http://www.ibm.com/developerworks/http://www.ibm.com/developerworks/cn/websphere/library/techarticles/1212_hart/1212_hart.htmlhttp://www.ibm.com/marketplace/cloud/developer-program/us/en-us?cm_sp=dwpremium-_-1212_hart-_-promobarhttps://www.ibm.com/marketplace/cloud/developer-program/us/en-us?cm_sp=dwpremium-_-1212_hart-_-adhttp://void%280%29/http://void%280%29/http://void%280%29/http://void%280%29/


2/12


http://www.ibm.com/developerworks/webspher e/library/techarticles/1212_hart/1212_hart.html 2

information later. If you need to scale this solution horizontally and add brokers to handle an increase in

throughput, then a database is the only reasonable option.

However, if a single in-memory cache is available to the request and response message flows regardless

of which brokers they are running in, then the request flow can store the correlation information in the

cache, and the reply flow can retrieve (and delete) it. Regardless of whether the reply message is routed

through the same broker as the original request, or through a different broker, the reply still ends up with

the correct client, as shown below:

Request/reply scenario with global cache

Other common scenarios for the global cache include maintaining an in-memory route table shared

between execution groups or brokers, or reducing latency to back-end systems by keeping a faade of

frequently queried data in the cache.

How does the global cache work?

The WebSphere Message Broker global cache is implemented using embedded WebSphere eXtreme

Scale (WebSphere XS) technology. By hosting WebSphere XS components, the JVMs embedded within

WebSphere Message Broker execution groups can collaborate to provide a cache. For a description of

WebSphere XS, see Product overviewin the WebSphere XS information center. Here are some of the

key components in the WebSphere XS topology:

Catalog server

Controls placement of data and monitors the health of containers.

Container server

A component embedded in the execution group that holds a subset of the cache data. Between them,

all container servers in the global cache host all of the cache data at least once.

Map

A data structure that maps keys to values. One map is the default map, but the global cache can have

several maps.

Each execution group can host a WebSphere XS catalog server, container server, or both. Additionally,

each execution group can make a client connection to the cache for use by message flows. The global

cache works out of the box, with default settings, and no configuration -- you just switch it on! You do not

need to install WebSphere XS alongside the broker, or any other additional components or products.

How do you control the scope of the cache?

The default scope of one cache is across one broker. To enable this, switch the broker-level policy
http://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1m1/topic/com.ibm.websphere.extremescale.doc/cxsprodoverview.html


3/12



property on the GlobalCache tab of Message Broker Explorer to Default and restart. This causes each

execution group to assume a role in the cache dynamically on startup. The first execution group to start

will be a catalog and container server, using the first four ports from the supplied port range (a port range

will have been generated for you, but you can modify this). For more details on the port range, see

Frequently asked questionsbelow. The second, third, and fourth execution groups (if present) will be

container servers, each using three ports from the range. Any execution groups beyond the fourth one

will not host cache components, but will connect as clients to the cache hosted in execution groups 1-4.

The diagram below shows the placement of servers, and the client connections, for a single-broker cache

with six execution groups:

Single-broker cache with default policy

You can extend the cache to multiple brokers by using a cache policy file. Three sample policy files areincluded in the product install, in the sample/globalcache directory. You can simply alter the policy file to

contain all the brokers you want to collaborate in a single cache, then point the broker-level cache policy

property at this file. Here is this setting in Message Broker Explorer:

Configuring a cache policy file in Message Broker Explorer


4/12



The file lets you nominate each broker to host zero, one, or two catalog servers, and the port range that

each broker should use for its cache components. The following diagram shows a two-broker cache, with

both brokers configured to contain catalog servers:

Two-broker cache controlled by policy file

You can also configure specific cache roles for each execution group, rather than having them

dynamically assigned on startup, by changing the broker cache policy property to none, which then

enables a set of detailed execution group properties. For details, see Cache topology considerations

below.

As the cached data is all hosted inside the execution group processes, the global cache is a better fit for

some solutions than others. For instance, storing transient correlation information or routing tables is


5/12



clearly a good use for this technology. Its suitability as a database faade depends largely on the amount

of data to be kept in the cache. Hosting several gigabytes of data in the JVM heaps of a small number of

execution groups is not recommended. Another consideration is the requirement for advanced features of

WebSphere XS. If you have identified a need to exploit APIs other than the simple ObjectMap interface,

or a requirement to customize the underlying WebSphere XS grid configuration XML files, then a

separate WebSphere XS or XC10 installation may be more appropriate. Connectivity to a remote a

WebSphere XS installation from WebSphere Message Broker is not covered by this article.

How do you interact with the cache?

The message flow developer has new, simple artifacts for working with the global cache, and is not

immediately aware of the underlying WebSphere XS technology or topology. Specifically, the Java

Compute node interface has a new MbGlobalMap object, which provides access to the global cache. This

object handles client connectivity to the global cache, and provides a number of methods for working with

maps in the cache. The methods available are similar to those you would find on regular Java maps.

Individual MbGlobalMapobjects are created by using a static getter on the MbGlobalMap class, which

acts as a factory mechanism. You can work with multiple MbGlobalMapobjects at the same time, and

create them either anonymously (which uses a predefined default map name under the covers inWebSphere XS), or with any map name of your choice. In the examples below, defaultMapwill work

with the system-defined default map within the global cache. myMapwill work with a map called myMap,

and will create this map if it does not already exist in the cache.

Sample MbGlobalMap objectsMbGlobalMap defaultMap = MbGlobalMap.getGlobalMap();MbGlobalMap myMap = MbGlobalMap.getGlobalMap(myMap);

How do you administer the cache?

Interactions with the global cache for a given execution group are tracked using resource statistics, and

an activity log that records the start of the catalog or container server in that execution group. It will then

record each put, get, update, remove or containsKey call made within that execution group, as well as

which flow and node made the call, which map and key were used, and details of any exceptions in

processing. The figure below shows the narrative of connecting to the global cache, checking whether an

entry exists, and then putting some data in the cache.

Example activity trace output

Since resource statistics and activity traces can provide only an isolated view from each execution group,

there is also the mqsicacheadmin command, which provides information about the entire global cache. It

has a number of subcommands. The listHostsand showPlacementcommands help validate the

condition of your global cache -- for example, does it cover the number of hosts you expect, does it have

the right number of containers, and are the shards spread evenly across these containers.


6/12



showMapSizesshows the overall size of your global cache, map by map, while clearGrid lets you clear

the data from a specific map, which is useful for purging data when a particular map gets too large.

Frequently asked questionsQ: Can one global cache be shared across multiple brokers?

A: Yes! Although the default cache policy provides a single-broker cache, you can use a cache policy file

to create a multi-broker cache. Three sample policy files are included in the product install. Simply alter

the policy file to contain all the brokers you want to collaborate in a single cache, and then point thebroker-level cache policy property at this file. The file lets you nominate each broker to host zero, one, or

two catalog servers, and the port range that each broker should use for its cache components. Ensure

that the listenerHost attribute for each broker in the policy file matches that broker's listenerHost property.

This is used, along with the broker name, by each broker to identify itself in the policy file.

Q: What happens if I shut down an execution group that hosts a catalog server?

A: If it is the only catalog server, then your cache is gone. However, if you have configured your cache to

have more than one catalog server, and they have successfully handshaked on startup, then the

remaining catalog server (or servers) will continue to run the cache without interruption. When you restart

the execution group, the catalog server will rejoin the topology.

Q: In a multi-broker cache, if one of my brokers fails, what happens to my data?

A: As long as you are running with multiple catalog servers on different machines, your data is still

available to the remaining brokers. Each piece of data in the cache has a primary version (primary

shard), and a replica. If you have multiple catalogs on different machines then the replica of any piece of

data is guaranteed to reside on a different machine from the primary. If the broker hosting the primary

shard of some data fails, the replica shard automatically becomes the primary, and a new replica is

created elsewhere. This behaviour is enabled by APAR IC88062, which guarantees that primary and

replica shards are placed on different machines. Without this APAR, it is possible to lose both shards if a

broker fails. If there is only one catalog server, or only one host containing catalog servers, then primary

and replica shards can be placed on the same machine.

Q: Is there a way to configure the cache to expire data after a certain amount of time?

A: Currently, there is no way to enable expiration in the embedded cache. The life cycle of all data must

be coded for in your message flows.

Q: Can I ensure that a specific execution group always hosts a catalog server?

A: Yes, by moving from a broker policy of default / file to none, and by setting each execution group's

cache properties individually. For more details on this topic, see Cache topology considerationsbelow.

Q: I'm trying out the global cache on my laptop and having problems when my IP address

changes. What is the solution?

A: For a single machine topology, you can change the listenerHost property of your broker (on the

broker's GlobalCache tab in Message Broker Explorer) to localhost. localhost will not work if you expand

the cache to cover multiple machines, but is suitable for this kind of simple, laptop-oriented DHCP usage.

Q: Does the global cache work for multi-instance brokers?

A: If an active broker fails, and a standby broker starts up to take its place on a different machine, the

catalogs and containers in that broker will fail to start (as they are binding to the wrong hostname). But all


7/12



execution groups will be able to make client connections to the global cache, assuming that a catalog is

still running in another broker. The original active broker will need to be restored in order for the cache

components (catalogs and containers) within that broker to restart and rejoin the cache.

Q: Is there a way to persist data to the file system or a database?

A: Currently there is no automatic way to do this with the global cache.

Q: Are interactions with the global cache transactional?

A: Each interaction with the global cache is a transaction in its own right. WebSphere XS pessimisticlocking is used during each action, and control is returned to the user only after that action has been

committed to the primary and replica versions of the data. But the global cache interactions are not

integrated with the message flow transaction, so if a message flow rolls back after some data has been

put into the cache, the data will not be removed. Keep this fact in mind when designing message flows,

and try to make cache puts, updates, and removes occur as late in the flow as possible. The Coordinated

Request Reply sample in the product demonstrates this technique storing correlation information into

the cache only after the flow's MQOutput node has successfully put a message.

Q: Can the global cache be accessed outside of a Java Compute node?

A: Yes. You can create static Java wrappers for common global cache tasks, such as getting a String

value from a given Map. You can then call these Java methods from within ESQL and the Mapping node.

Making the cache available natively from across the programming interfaces is the long-term goal.

Q: Are there any implications for the size of execution group processes when using the global

cache?

A: With a catalog or container server enabled, the heap size of the JVM within an execution group will

increase by at least 30 MB. There is no impact on the performance of flows within that execution group

as a result of hosting a cache component, but all of the data in the global cache will be hosted within your

execution group processes, and the size of these processes will grow accordingly. If you create flows thatcontinually write to the cache and never remove data, the size of your processes will keep growing until

they cause out-of-memory errors. You should consider how much data is likely to be placed in the cache,

and refer to the WebSphere XS product documentation for guidance on JVM heap size settings.

Q: Why doesn't the default cache policy allow for more than one catalog server?

A: Although running with multiple catalog servers is recommended, there is some overhead involved.

When you have multiple catalogs, ensure that a majority of them are started at around the same time,

and allow 60-90 seconds for them to handshake before the cache becomes available. A default single-

broker cache with one catalog still provides sharing of data across execution groups, and starts up

quickly.

Q: Do I need to restart all my brokers if I modify the cache policy file?

A: It depends on what you have modified. If you have added, removed or modified a broker that hosts a

catalog server, then restart all of the brokers. All of the catalog servers need to know about each other

before starting, and adding a new catalog server dynamically whilst the others are still running is not

possible. However, if you add a broker that does not contain a catalog server, then you only need to

restart this one broker. The container servers and client connections within this broker will connect to the

grid using the information already present in the policy file, and the other components do not need to be


8/12



made aware of this change.

Q: Why is a port range required, and how large should it be?

A: The catalog server and container server components within the cache all require a number of ports to

communicate. Execution groups that are hosting catalog servers (or catalogs and containers) require four

ports. Execution groups that host only container servers require three ports. Execution groups that act

purely as clients do not require any ports. WebSphere Message Broker requires that you provide a range

of at least 20 ports. Currently, a maximum of either 13 ports (for the default cache policy) or 14 ports (for

a two-catalog broker defined via the policy file) are actually used per broker. The additional ports are

reserved for future use.

Q: Does every container have a full copy of the cache?

A: No. Each container hosts a subset of data in the cache. Each piece of data has a primary copy and a

replica. The WebSphere XS components in the cache ensure that those primary and replica copies

(shards) are spread across the available containers.

Cache topology considerations

What is cache topology?Cache topology is the set of catalog servers, containers, and client connections that collaborate to form a

global cache. When using the default policy, the first execution group to start will perform the role of

catalog server and container server (call this Role1). The next three execution groups to start perform the

role of container servers (Role2, 3, and 4). No other execution groups will host catalogs or containers, but

all execution groups (including those performing Roles1, 2, 3, and 4) host client connections to the global

cache. When you restart the broker, the execution groups may start in a different order, in which case

different execution groups might perform Roles1, 2, 3, and 4. The cache topology still contains one

catalog server, up to four containers, and multiple clients, but the distribution of those roles across your

execution groups will vary.

How does a cache policy file help with this?

The policy file lets you define the shape of your cache topology across one or more brokers. If you create

a policy file with two brokers, each of which is to host one catalog server, then you are actually just joining

together two default caches. Each broker will have the same Roles1, 2, 3, and 4 described above but

they all know they are part of a multiple-catalog cache, and will set up their properties accordingly.

However, you can also use the policy file to define a single-broker cache with two catalog servers. In this

case, Role2 described above becomes a second catalog server, and a container server.

You can also specify that a particular broker should contain zero catalog servers, which is useful in acache that spans multiple brokers. A small, prime number of catalog servers is ideal for optimum

availability and performance. So in a cache spread across six brokers, you might want only three of them

(for example) to host catalogs. For a discussion of when you have to restart brokers after a change to the

policy file, see Frequently asked questionsabove.

When should you use policy none and control each execution group

individually?

When you want to know exactly which role each execution group is playing in the topology, and have


9/12



this fixed across broker restarts.

When you are using the cache heavily and need to tune the JVM heap size, and therefore need to know

which execution groups to tune.

When you want to isolate the catalog server, so that you can manage that execution group's life cycle

individually.

When you are using the cache heavily, and do not want the catalog server coexisting in a JVM with a

container, since the work performed by the catalog could affect the performance of that container.

When you want to create more container servers or catalog servers in a broker than any of the policy

options provide.

When you want to configure each port individually, rather than having the broker pick ports from a range.

Having set the policy to none, what do you do next?

When you choose the broker policy setting of none, a set of execution group properties that were

previously read-only become available. These are visible in Message Broker Explorer on the

GlobalCache tab of the execution group's properties, and also from the command line via the

ComIbmCacheManagerresource within the execution group:

mqsireportproperties -e -o ComIbmCacheManager -r

When you switch from a policy (either the default or a policy file) to none, the most recent values of these

properties that were dynamically set by the policy are retained as a starting point for customization.

Customizing properties for execution groups that you want to not host catalog servers is relatively

straightforward:

Three ports are required listenerPort, haManagerPort, and jmxServicePort.

You can disable JMX (set enableJMX to false), but the mqsicacheadmincommand will then be unableto reach components in this execution group.

connectionEndPointsmust be set to a comma separated list of catalog servers, with each catalog

server described in the format listenerHost:listenerPort. This property is used by the container

server (if enabled) in this execution group, and by the client connection, to communicate with the cache.

The catalogClusterEndPointsproperty does not need to be set.

Customizing properties for execution groups that you want to host catalog servers is a little more difficult.

In addition to the steps above, you must also set the catalogClusterEndPointsproperty:

Each execution group that is to host a catalog server must have the same value for

catalogClusterEndPoints.

This property contains details of not only this execution group's catalog, but also every other catalog

server in the topology.

Each endpoint is in the format ServerName:listenerHost:CatalogPeerPort:haManagerPort.

The CatalogPeerPortis a fourth port, for catalog servers only, and is not referenced elsewhere in the

properties.


10/12


http://www.i bm .com /developer wor ks/w ebspher e/l ibr ar y/techar ti cl es/1212_har t/1212_har t.html 10

Resources

WebSphere Message Broker resources

Global cache overview

Topic in the WebSphere Message Broker V8.0.0.1 information center.

Product overview

Topic in the WebSphere eXtreme Scale information center.

Parameter values for the cachemanager component

Topic in the WebSphere Message Broker V8.0.0.1 information center.

WebSphere Message Broker V8 information center

A single Web portal to all WebSphere Message Broker V8 documentation,

with conceptual, task, and reference information on installing, configuring,

Dig deeper into WebSphere ondeveloperWorks

Overview

New to WebSphere

Technical library (tutorials and more)

Forums

Community

Downloads

Products

Events

listenerHostand haManagerPortare the same values as specified in this execution group's

properties.

The server name is BrokerName_listenerHost_listenerPort. To give an example:

You want this execution group to host a catalog server, and you have set the listenerPortto 2800,

the haManagerPortto 2801, and the jmxServicePortto 2802.

You want the fourth port (CatalogPeerPort) to be 2803.

The broker is called MB8BROKER.

The listenerHost is set to mybrokerbox.

The catalogClusterEndPointfor this catalog server will be

MB8BROKER_mybrokerbox_2800:mybrokerbox:2803:2801

catalogClusterEndPointsfor this execution group need to be set to a comma-separated list of the

end point above, and all other catalog servers.

Due to the complexity of setting the catalog server properties, here are some alternative approaches forsetting individual execution group properties:

Start with a cache policy. Bring up execution groups in the correct order based on the roles you want

them to take. Switch to none to fix these roles. You can then also tweak the settings without having to

craft them from scratch (for example, switch enableContainerServiceto false to achieve an isolated

catalog server).

Again, start with a cache policy. Based on the discussion above about roles within the topology, you can

consider each set of execution group properties to constitute a role definition, which is not tightly

coupled to any one execution group. Use mqsireportpropertiesto create a snapshot of these role

definitions. You can now switch to none and use mqsichangepropertiesto assign these roles to the

specific execution groups. For example, EG1 may have taken Role1 under the default policy, but you

actually want a dedicated execution group called CATCONT1 to perform that role. You can report EG1's

properties and use those precise values to set up CATCONT1. If you use this approach, make sure that

you revisit EG1's properties to ensure that it now has the correct role (even if that just involves setting

enableCatalogServiceand enableContainerServiceto false).
http://www.ibm.com/developerworks/websphere/find/events/http://www.ibm.com/developerworks/websphere/products/http://www.ibm.com/developerworks/websphere/downloads/http://www.ibm.com/developerworks/websphere/community/https://www.ibm.com/developerworks/community/forums/html/category?id=33333333-0000-0000-0000-000000000009http://www.ibm.com/developerworks/websphere/library/http://www.ibm.com/developerworks/websphere/newto/http://www.ibm.com/developerworks/websphere/http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=/com.ibm.etools.msgbroker.helphome.doc/help_home_msgbroker.htmhttp://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/bn23785_.htmhttp://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1m1/topic/com.ibm.websphere.extremescale.doc/cxsprodoverview.htmlhttp://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/topic/com.ibm.etools.mft.doc/bn23730_.htm


11/12


12/12


changing and increasingly complex business environment.

WebSphere-related articles on developerWorks

Over 3000 edited and categorized articles on WebSphere and related

technologies by top practitioners and consultants inside and outside IBM.

Search for what you need.

developerWorks WebSphere weekly newsletter

The developerWorks newsletter gives you the latest articles and information

only on those topics that interest you. In addition to WebSphere, you can

select from Java, Linux, Open source, Rational, SOA, Web services, and

other topics. Subscribe now and design your custom mailing.

WebSphere-related books from IBM Press

Convenient online ordering through Barnes & Noble.

WebSphere-related events

Conferences, trade shows, Webcasts, and other events around the world of

interest to WebSphere developers.

developerWorks resources

Trial downloads for IBM software products

No-charge trial downloads for selected IBM DB2, Lotus, Rational,

Tivoli, and WebSphere products.

developerWorks business process management developer resources

BPM how-to articles, downloads, tutorials, education, product info, and other

resources to help you model, assemble, deploy, and manage business

processes.

developerWorks blogs

Join a conversation with developerWorks users and authors, and IBM

editors and developers.

developerWorks tech briefings

Free technical sessions by IBM experts to accelerate your learning curve

and help you succeed in your most challenging software projects. Sessions

range from one-hour virtual briefings to half-day and full-day live sessions in

cities worldwide.

developerWorks podcasts

Listen to interesting and offbeat interviews and discussions with software

innovators.

developerWorks on Twitter

Check out recent Twitter messages and URLs.

IBM Education Assistant

A collection of multimedia educational modules that will help you better

understand IBM software products and use them more effectively to meet

your business requirements.
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsphttp://twitter.com/developerworkshttp://www.ibm.com/developerworks/podcast/http://www.ibm.com/developerworks/briefings/http://www.ibm.com/developerworks/blogshttp://www.ibm.com/developerworks/bpm/http://www.ibm.com/developerworks/downloads/http://www.ibm.com/developerworks/views/websphere/events.jsphttp://www.redbooks.ibm.com/redbooks.nsf/allibmpress?SearchView&query=webspherehttps://www.ibm.com/developerworks/newsletter/http://www.ibm.com/developerworks/views/websphere/libraryview.jsp?type_by=Articles

introduction to the websphere message broker global cache

Documents