wtu 2014 ibm web sphere mq for zos - shared queues
DESCRIPTION
What are shared queues in IBM MQ, and diving into the techincal details of configuring your setup.TRANSCRIPT
© 2014 IBM Corporation
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
2
3
Agenda
• What are Shared Queues?
• Large messages with DB2
• Shared Message Data Sets (SMDS)
• Structures – Persistence and recovery
5
Shared queues
SharedQueueCoupling Facility (CF)
Application Application
QMgr QMgr
Same Sysplex
7
Queue-sharing groups (QSGs)
Queue-Sharing Group
Shared Objects
Msg data> 63KB
DB2 Data Sharing Group
CF
Shared Queues
QMGR
Private Queues
Private Objects
ChannelInitiator
QMGR
ChannelInitiator
QMGR
Private Queues
Private Objects
ChannelInitiator
QMGR
ChannelInitiator
DB2A DB2B DB2C
Msg data> 63KB
Msg data> 63KB
Msg data> 63KB
11
Shared Message Data Sets (SMDS)
Queue-Sharing Group
Shared Objects
DB2 Data Sharing Group
CF
Shared Queues
QMGR
Private Queues
Private Objects
ChannelInitiator
QMGR
ChannelInitiator
QMGR
Private Queues
Private Objects
ChannelInitiator
QMGR
ChannelInitiator
DB2A DB2B DB2C
Msg data> 63KB
SMDS SMDS
V7.1
13
CF structures for shared queues
{Administration
structure
Applicationstructures
Information for unit-of-work recovery
Coupling Facility
Administrationstructure
Applicationstructures
Queue Queue Queue
Information for unit-of-work recovery
{
Structuresfor QSG 1
Structuresfor QSG 2
Queue Queue Queue x 512
x 63
15
Creating CF structures and shared queues
• Define a structure to z/OS by updating the CFRM policy:– The structure is known to MQ by its 12-character str-name.
– The structure is known to z/OS by the 16-character name formed by:• Application structures: qsg-name || str-name
• Administration structures: qsg-name || CSQ_ADMIN
• Application structures are defined to MQ using the DEFINE CFSTRUCT command on any queue manager in the QSG
• Shared queues are defined using the DEFINE QLOCAL command on any queue manager in the QSG:– DEFINE QLOCAL(queue-name) QSGDISP(SHARED) CFSTRUCT(str-name)
• z/OS allocates the structure in the CF when required (first use)
• MQ creates the queue when required (first use)
17
Large shared queue messages (using SMDS)
QM1
Ptr to 100K message
QM1SMDS
QM2SMDS
QM2
APPMQPUT
APPMQGET
1
2 3
4
Shared queue
V7.1
• One SMDS per queue manager per CF structure
• Each queue manager only writes large messages to its own SMDS
• Queue managers can get messages from any queue manager’s SMDS
19
SMDS performance improvement
• Tests show comparable CPU savings, making SMDS a more usable feature for managing your CF storage
• SMDS per CF structure provides better scaling than DB2 BLOB storage
1 2 3 4 5 6 7 8 9 10
0
50
100
150
200
250
300
350
400
3 LPAR Test - DB2
64KB Non-Persistent Messages In-Syncpoint - DB2
NP SIS Scaling – 3 qmgr
NP SIS Scaling – 6 qmgr
NP SIS Scaling – 9 qmgr
Queue Pairs
Tra
nsa
ctio
ns
/ S
eco
nd
1 2 3 4 5 6 7 8 9 10
0
1000
2000
3000
4000
5000
6000
7000
3 LPAR Test - SMDS
64KB Non-Persistent Messages In-Syncpoint - SMDS
NP SIS Scaling – 3 qmgr
NP SIS Scaling – 6 qmgr
NP SIS Scaling – 9 qmgr
Queue Pairs
Tra
nsa
ctio
ns
/ S
eco
nd
21
Creating a Shared Message Data Set
• SMDS is defined as a VSAM data set using DEFINE CLUSTER– Requires LINEAR option
– Control interval size must be 4096, which is the default for linear data sets
– Requires SHAREOPTIONS(2 3), that allows one queue manager to write and other queue managers to read at the same time
– If maximum size may need to exceed 4GB, requires SMS data class which has VSAM extended addressability attribute
– If automatic expansion is to be supported, requires an appropriate secondary space allocation, although a default of 20% will be used if an expansion attempt fails because of no secondary allocation
• Can optionally be pre-formatted, for example using CSQJUFMT– Otherwise formatted automatically when first opened
23
Access to Shared Message Data Sets
• Shared message data sets must be on shared direct access storage accessible to all queue managers in the QSG.
• Normal running:– Queue manager opens own data set read/write.
• Requires UPDATE access to own data set.
– Queue manager opens other data sets read-only.• Requires READ access to all other data sets.
• Media recovery processing:– Queue manager performing recovery opens own data set and all other
data sets for read/write access.• Requires UPDATE access to all data sets.
24
Shared Message Data Set capacity considerations
• Each shared message data set only contains data for large messages written by its owning queue manager
• Message size calculation:– Each stored message includes standard headers (usually 352 bytes)– Each message is stored as one or more message blocks– Each message block is stored in a range of consecutive 4KB pages on the
data set, with a very small header (32 bytes)– Approximate data set space required per large message, in bytes, is given by
size of message plus the header rounded up to the next 4KB
• Multiply by the maximum anticipated backlog of messages written by that queue manager (plus some safety margin) to estimate the required data set size
25
Shared Message Data Set expansion
• Data set can be automatically expanded when necessary– Normally configured by the DSEXPAND(YES|NO) option on the CFSTRUCT
object, which specifies the default option for the data set group.– Can also be overridden for individual data sets using ALTER SMDS with the DSEXPAND option
• Expansion attempt is automatically triggered when 90% full– If no secondary allocation was specified a VSAM error message will appear,
but the queue manager will retry using a default secondary allocation of 20% of the existing size
– If expansion fails because there is not enough space available the queue manager sets DSEXPAND(NO) to prevent further attempts. The operator can use ALTER SMDS to restore DSEXPAND(YES) once the problem has been resolved
– If the maximum number of extents are reached the data set cannot be expanded further. However, it could be marked unavailable then copied to a larger data set, which is subsequently renamed to the original name
26
Selecting which messages to offload
• Messages too large for CF entry (> 63KB) are always offloaded
• Other messages may be selectively offloaded using offload rules– On each CFSTRUCT definition, you can specify 3 off-load rules
– Each rule specifies:
• Message size in KB – OFFLDnSZ(size)
• Structure usage threshold – OFFLDnTH(percentage)where n = 1, 2, 3.
• Data for new messages exceeding the specified size is offloaded (as for a large message) when the structure usage exceeds the specified threshold
• Default rules are provided which should be useful in most cases
• Rules can be set to dummy values if not required
Offloading CF structure size Message size Number of messages
No 100GB 63KB 1.25M
Yes 100GB Any 140M
28
Storage benefits of offloading
• 1GB structure
• 20KB messages being put
~ 35,000 messages in CF
Thresholds for % of CF structure full
• ~ 320,000 messages using offloading versus ~ 50,000 without offloading
~ 5,000messages
~ 140,000offloadedmessages
~ 140,000offloadedmessages
Message sizes (default rules)
> 63KB > 32KB > 4KB > 0KB
70% 80% 90%0%
30
SCM Storage (Flash) – Planned Emergency Storage
• z/OS 2.1 and zEC12 GA2 introduce Coupling Facility Flash storage
– Additional capacity for maintenance windows or consumer outages
• Consider CFSTRUCT OFFLOAD rules:
– OFFLD1TH(70), OFFLD1SZ(32K)• Offload messages larger than 32KB if the structure is 70% full
– OFFLD2TH(80), OFFLD2SZ(4K)• Offload messages larger than 4KB if the structure is 80% full
– OFFLD3TH(90), OFFLD3SZ(0K)• Offload all message data if the structure is 90% full
• Progressively smaller messages written to SMDS as the structure fills
• At 90% full the queue manager stores the minimum data per message to squeeze as many message references as possible into the remaining storage
• CF Flash algorithm also starts moving the middle of the queue out to flash storage, keeping the faster 'real' storage for messages most likely to be got next
32
SCM Storage (Flash) – Maximum speed
• Quicker to access messages stored in the CF than in SMDS– Keep high performance messages in the CF
•CFSTRUCT OFFLOAD rules disabled using OFFLD1SZ(64K)– Maybe one rule to reduce large message threshold below 63KB
• At 90% full the CF Flash algorithm starts moving the middle of the queue out to flash storage, keeping the faster 'real' storage for messages most likely to be got next.
• As messages are got and deleted the CF Flash algorithm attempts to pre-fetch the next messages from Flash storage they are rapidly available for MQGET.
• In this scenario the Flash storage acts like an extension to 'real' CF structure storage. However it is consumed more rapidly in this scenario since all small message data is stored in it.
34
SCM Storage (Flash) – Comparison
Scenario MessageSize
TotalMessages
# Messagesin 4GB 'real‘
SMDSSpace
# Messages in200GB Flash
Augmented(limit 30GB)
No SMDSNo Flash
1KB 3M 3M
4KB 900,000 900,000
16KB 250,000 250,000
SMDSNo Flash
1KB 3.2M 3.2M 800MB
4KB 1.8M 1.8M 5GB
16KB 1.3M 1.3M 20GB
EmergencyScenario
1KB 190M 2M 270GB 190M 30GB
4KB 190M 600,000 850GB 190M 30GB
16KB 190M 150,000 3TB 190M 30GB
SpeedScenario
1KB 150M 2M 150M 26GB
4KB 48M 600,000 48M 8GB
16KB 12M 150,000 12M 2GB
36
Failure and persistence
Sharedqueues
Coupling Facility failure
Queuemanager
Privatequeues
Queuemanager
Privatequeues
Nonpersistentmessages onprivate queuesOK (kept)
Messages onshared queuesOK (kept)
Queue manager failure
Persistent messages onshared queuesrestored from log
Queuemanager
Privatequeues
Queuemanager
Privatequeues
Queuemanager
Privatequeues
Queuemanager
Privatequeues
Queuemanager
Privatequeues
Nonpersistentmessages onprivate queueslost (deleted)
Sharedqueues
Nonpersistent messages onshared queueslost (deleted)
38
Administration structure recovery
• In a disaster recovery situation (e.g. loss of CF)– Prior to version 7.0.1
• Each queue manager rebuilds its own admin structure entries• Therefore, you need to start all queue managers to rebuild admin structure• Once recovered, application structures can be recovered
– From version 7.0.1• Active queue managers notice if other queue managers don’t have entries
in admin structure and initiate rebuild of admin structure entries on their behalf
• Application structures can be recovered before all queue managers are restarted
• A version 7.0.1 or higher, queue managers can recover entries on behalf of any queue manager at the same or lower version of MQ– Not all queue managers in the QSG need to be at V7.0.1, or higher
40
CF loss of connectivity
CF
QM2
QM1
QM3
Pre-V7.1 Queue Managers
A failure of the Coupling Facility is most likely going to be presented to connectors as a Loss of Connectivity
Prior to V7.1 if a queue manager receives a loss of connectivity it will terminate.
If a Coupling Facility failure occurs this results in an outage for the entire QSG, unless protected by CF Duplexing
41
CF loss of connectivity tolerance
CF2V7.1 or later queue managers will not terminate. They will automatically attempt to re-establish access to the affected structures.
CF1
QM2
QM1
QM3
V7.1+ Queue Managers
QM2
QM1
QM3
V7.1+ Queue Managers
If a total loss of connectivity occurs the queue managers can automatically recover (RECOVER CFSTRUCT) the structures that were on the failed CF in to an alternative CF if one is available).
V7.1
43
CF loss of connectivity tolerance
CF2CF1
QM2
QM1
QM3
V7.1+ Queue Managers
If a partial loss of connectivity occurs a System Managed Rebuild is automatically initiated by the queue managers to rebuild the structures in to a more available CF. This means both persistent and non-persistent messages are retained.
V7.1
45
CF admin structure loss of connectivity tolerance
• Queue managers will tolerate loss of connectivity to the admin structure without terminating if
– The QMGR CFCONLOS attribute is set to TOLERATE
– All the queue managers in the QSG are at version 7.1 or later
• All queue managers in the QSG will disconnect from the admin structure, then attempt to reconnect and rebuild their own admin structure data
• If a queue manager cannot reconnect to the admin structure, for example because there is no CF available with better connectivity, some shared queue operations remain unavailable until the queue manager can successfully reconnect to the admin structure and rebuild its admin structure data
• The queue manager will automatically reconnect to the admin structure when a suitable CF becomes available on the system
• Failure to connect to the admin structure during queue manager start-up is not tolerated, regardless of the value of CFCONLOS
47
CF structure definition considerationsMQ CFSTRUCT definition CFRM policy structure definition
CFSTRUCT(TEST1) STRUCTURE NAME(SQ27TEST1)CFLEVEL(5) SIZE(50000)CFCONLOS(TOLERATE) INITSIZE(20000)RECAUTO(YES) DUPLEX(DISABLED)OFFLOAD(SMDS) ALLOWAUTOALT(YES)
PREFLIST(P5CF01,P5CF02)
• If using CFCONLOS(TOLERATE) you should also consider:– RECAUTO – automatic recovery of application structures– PREFLIST – to allow the structure to be allocated in multiple Coupling Facilities
• ALLOWAUTOALT(YES) – Enables CF to adjust entry/element ratio, and also automatically resize structure up to
a maximum SIZE value – MINSIZE can be set to automatically adjust the structure size down
• CF duplexing can be used to duplex MQ structures– Makes most types of failures transparent to MQ
For Additional Information
IBM Traininghttp://www.ibm.com/training
IBM WebSpherehttp://www.ibm.com/software/websphere/
http://www.ibm.com/software/products/ibm-mq
IBM developerWorkshttp://www.ibm.com/developerworks/websphere
https://www.ibm.com/developerworks/community/blogs/messaging
WebSphere forums and communityhttp://www.ibm.com/developerworks/websphere/community/
IBM MQ Sessions this week10:30 - 12:00 13:15 - 14:15 14:30 - 15:30 16:00 - 17:00 17:15 - 18:15
Tu
esday
Opening General Session- IBM Digital Experience and WebSphere Technical University
Session A31: IBM MQ CHLAUTH rules – with MQ V8 updates
Speaker: Morag HughsonRoom 02
Session A4: WebSphere MQ for z/OS: Performance and Accounting
Speaker: Alex RossRoom 8
Session I26: DataPower-MQ Connectivity Deep Dive (Theory)
Speaker: Robin WileyRoom 27
Session Z1: WebSphere MQ for z/OS V8: Latest Features Deep Dive
Speaker: Damon CrossRoom 6
9:00 - 10:00 10:30 - 11:30 11:45 - 12:45 14:00 - 15:00 15:15 - 16:15 16:45 - 17:45
Wed
nesd
ay
Session Z5: WebSphere MQ for z/OS: Security
Speaker: Damon CrossRoom 02
Session A21: What's New in IBM Messaging
Speaker: Morag HughsonRoom 8
Session C7: Messaging in the Cloud with IBM MQ Light and IBM Bluemix
Speaker: Rob NicholsonRoom 27
Session A17: Managing work-loads, scaling and availability with IBM MQ clusters
Speaker: David WareRoom 6
Lab IL5: DataPower-MQ Connectivity Deep Dive (Hands-On)
Speaker: Robin WileyRoom 7b
Session A9: WebSphere MQ for z/OS: The Inside Story
Speaker: Damon CrossRoom 6
Th
ursd
ay
Session A35: How to Develop Responsive Applications with IBM MQ Light
Speaker: Rob NicholsonRoom 27
Session A22: New IBM MQ V8 Security Features
Speaker: Morag HughsonRoom 01
Session A3: WebSphere MQ for z/OS: Shared Queues
Speaker: Alex RossRoom 6
Session A18: Using Publish /Subscribe with IBM MQ
Speaker: David WareRoom 27
Frid
ay
Lab AL6: Developing a First Application with IBM WebSphere MQ Light
Speakers: Robert Nicholson, Alex RossRoom 7b
Session A16: Using IBM MQ Pub/Sub in an MQ network
Speaker: David WareRoom 6
© 2014 IBM Corporation 50
Questions?