websphere application server for z/os v8 hidden …...hidden gems in the service stream but wait,...

WebSphere Application Server for z/OS V8

Hidden Gems 3: Even More Great But Little Known Features

of WebSphere Application Server on z/OS

This document can be found on the web at:www.ibm.com/support/techdocs

Search for document number WP101992 under the category of "White Papers"

Version Date: October 4, 2011See "Document change history" on page 17 for a description of the changes in this version of the document

IBM Software GroupApplication and Integration Middleware Software

Written by David Follis IBM Poughkeepsie

[email protected]

mailto:[email protected]

WP101992 – WebSphere Application Server V8 – Hidden Gems

Many thanks go to ...The zRuntime team for getting all this work done and to the WSC, especially Don Bagwell, Mike

Loos, and Mike Cox plus assorted customers who suggested some of these improvements

© 2009, IBM Software Group, Application and Integration Middleware Software, Poughkeepsie, NY - 2 - Version Date: Tuesday, October 04, 2011


Introduction ........................................................................................................................................... 4 Hidden Gems in the Service Stream .................................................................................................... 4

More About Shared Library Region Size .............................................................................................. 4 Options for the Error Logstream Browser .............................................................................................. 5 Improved Diagnostics for Error Logstream Problems ............................................................................ 6

Hidden Gems in Version 8 .................................................................................................................... 6 Routing Java Diagnostic Commands .................................................................................................... 6 Form Feed Controls .............................................................................................................................. 7 WLM AE_SPREADMIN Option ............................................................................................................. 7 Displaying Server State ......................................................................................................................... 8 WLM Delay Monitoring Miscellaneous States ....................................................................................... 9 Improved Message Routing ................................................................................................................. 9 Eliminate Redundant Tracing of Logged Messages ............................................................................ 11 SMF 120-9 Affinity Information ............................................................................................................ 12 Optimized Load Modules ..................................................................................................................... 13 Tagdata in the 'ps' and Display,OMVS,PID Command ........................................................................ 14 Thread Hang Recovery Diagnostic Improvements .............................................................................. 15 Handling Work With Affinity During a Timeout Delay ........................................................................... 20

Document change history ................................................................................................................... 22



IntroductionFirst there was WebSphere Application Server for z/OS V6 Hidden Gems (WP101138). Then came WP101464, also known as WebSphere Application Server for z/OS V7 Hidden Gems. Naturally, like that third “Back to the Future” move you knew was coming, we have Hidden Gems for Version 8. Will there be, Indiana Jones-like, a fourth edition? We'll see. But for now get some popcorn, settle back and relax and we will take a stroll through some of the lesser known but perhaps quite important features delivered in WebSphere Application Server for z/OS Version 8!

Hidden Gems in the Service StreamBut wait, before we dive into Version 8, all has not been quiet in the service stream. A few things we think qualify as 'gems' have turned up as part of a few APARs delivered for V6.1 and V7. Let's take a look....

More About Shared Library Region Size Most of the technical details for this one are already covered in the whitepaper WP101320. You should go read it. I'll wait here... dum-de-dum-de-dum..what? Back already? Hmm... you read pretty fast. Perhaps I should summarize in case you missed something.

• USS allows you specify an area of high-private storage to be shared between address spaces for modules loaded from the USS File System.

• Its the same in all regions, but for a 31 bit WAS server a large shared region size might make it impossible to get the JVM heap size you wanted.

• USS introduced _BPXK_DISABLE_SHLIB in OA33516 to let you not have a shared library region in a particular address space (like a 31 bit WAS servant region)

• And WP101320 told you how to set this up and then verify it was working with an SVCDUMP and IPCS.

Why rehash this here? Well, we made the verification part a lot easier. In PM36368 (V6.1) and PM32677 (V7) we updated the BBOO0341I message to tell you what happened.You don't remember BBOO0341I? You can go read about it at... oh never mind. The message previously looked like this:

BBOO0341I VARIOUS RESOURCE MONITORING DATA: (64):():():():():():():():():()The '64' inside the first parenthesis is the shared library region size configured for USS. All those extra parenthesis were place-holders in case we came up with other information we wanted to include. Well, now with this improvement the message might look like this:

BBOO0341I VARIOUS RESOURCE MONITORING DATA: (64):(67108864):():():():():():():():()The second fill in indicates the shared library region size for this address space (in bytes instead of megabytes). If you set the BPXK variable to disable the shared library region then the second fill-in will be zero.



Options for the Error Logstream BrowserThe server 'error log' is essentially a place for the server to write messages that aren't important enough to be WTOs. By default this log is directed to the SYSOUT DD card in the JCL procedure for the controller or servant region. However, you can also use the ras_log_logstreamName environment variable to direct the server to send this output to a predefined z/OS System Logger logstream. Why would you want to do that? The SYSOUT DD card can be directed nearly anywhere, but most customers let it just go to the job output in the spool (SYSOUT=*) or point it to a file in the USS File System. The file system approach is popular because it makes the output available to application developers in a manner that is more familiar to them from distributed systems than using SDSF to look at job output. But spool and file systems can fill up and require automation of one sort or another to manage. Furthermore, each process has its own job output or file. That means every controller and each servant region creates separate output. Sometimes that makes it easier to see what is going on in each process, but in debugging a problem it can sometimes be nice to see the whole server as one unit. This leads you to the logstream. Logstreams can be defined with automatic archiving and retention policies. Then z/OS manages it for you. Furthermore, several processes or even several servers can share the same logstream, allowing output to flow together in a time-ordered fashion which, sometimes, can make it easier to see what is going on.The biggest complaint about using logstreams instead of the SYSOUT DD card is that the logstream is difficult to look at. You can not just browse it like a file. Some piece of code has to call the System Logger APIs to extract the data from the stream. To help with this situation WebSphere Application Server ships a REXX exec called “BBORBLOG”. You can use this exec under ISPF to extract the data from the logstream. The exec places the data in a sequential dataset and places you in ISPF Browse for that dataset.So far so good, right? We even have an option to control whether the lines are automatically wrapped at 80 characters or are allowed to stray as far to the right as they need to. Usually the auto-wrapped version is more readable, but sometimes the one-line-per-record approach makes it easier to find things.But alas there were problems. We even took a FITS requirement (MR0924072223 for those keeping score at home). The requirement complained about how the dataset was allocated. We had chosen to hard-code the file name to be whatever the TSO Profile's “prefix” value was set to followed by the name of the logstream you asked us to browse, every time you ran the browser. Might be nice if you could control the file name where the output went. To help with this, as part of PK91010 which addressed some other problems in the logstream browser, we added two new keywords to the syntax for this utility. The keywords are reclen and outdsn. The outdsn keyword allows you to specify the non-fully qualified dataset name that will be used for the contents of the stream.The reclen keyword allows you to specify the logical record length for the output dataset the exec is going to allocate. If you don't specify reclen then the logical record length used by default is either 80 or 4096 depending on whether or not you asked us to wrap the output. You might want to set a value beyond 4096 if you choose to not wrap the output and end up with lines that are very very very long. The largest value you can specify is 32752.This change shipped in V6.1 and V7. You can read more about it in the Information Center here:http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.zseries.doc/info/zseries/ae/ttrb_usebrw.html


http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.zseries.doc/info/zseries/ae/ttrb_usebrw.html

http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.zseries.doc/info/zseries/ae/ttrb_usebrw.html


Improved Diagnostics for Error Logstream ProblemsWhile we are on the subject of logstreams we should probably mention PK85438. This APAR involved the code inside the application server that writes to the error logstream. This code calls the z/OS System Logger API to write data. Like all good z/OS system services, there are return codes to tell the caller if the service worked. Like a good user of z/OS system services, WebSphere checked the return code. Unfortunately we panicked a bit if we got back something besides zero.Some non-zero return codes indicate serious problems and the code reacted correctly by giving up on the logstream and falling back to just writing to the SYSOUT DD card. Unfortunately, the code wasn't very good at telling you what the non-zero return code was to allow you to correct the problem. Furthermore, some return codes are just transient problems. For example, the z/OS System Logger might be overrun with messages and can not find resources to deal with this message at this time. Or perhaps the message we tried to write is simply too big for the logstream (based on how the logstream was defined). Neither of these problems warrant giving up completely on the logstream.And thus we took the APAR PK85438. This APAR changes this code to do a better job of reporting the nature of the errors encountered with the logstream with the BBOO0082E message. Furthermore, any message we fail trying to write to the logstream will be written to the SYSOUT DD card. So what makes this a 'gem'? Well... as in editions of this paper for previous releases we also try to point out areas where we have learned from experience and improved the quality of the code to handle errors better and to make it easier to diagnose problems.

Hidden Gems in Version 8On to Version 8!

Routing Java Diagnostic CommandsThis item was done in response to a FITS requirement (MR121708640). To understand the change we first need to backtrack a little. Back in Version 7 we introduced new Modify commands to gather diagnostic information from the JVM. These are covered in the V7 Hidden Gems paper, but basically we let you trigger a JAVACORE, a HEAPDUMP, or a JAVA TDUMP from the MVS console. You used the Modify command interface to tell us what you wanted and we would trigger the appropriate dump in the controller and in all the servants (well, we didn't TDUMP the controller for some reason). If you only have one servant region, then this is just fine. You get the doc you wanted from the server. But if you have more than one servant, or if you have a lot of servants, then you get a lot of doc collected that you probably don't want. In most cases where you need a heapdump or a javacore you probably know which servant region you want the dump from. It would be nice if you could tell us.In version 8 we added an option to these commands to specify the ASID of the servant region you want dumped. Just add a “ASIDX=” after the command with the appropriate ASID (in hex) of the servant region you want to dump. For example:MODIFY server,JAVACORE,ASIDX=F4



This command will take a javacore of the servant region whose (hex) ASID is 'F4'. You can also specify the controller's ASIDX which will dump just the controller, no servant regions. And now you can request a TDUMP from the controller by specifying the controllers ASIDX.

Form Feed ControlsThis is another improvement to a hidden gem from previous releases. Way back in the V6.1 Hidden Gems we talked about using the JES2 SEGMENT DD keyword to cause job output to spin allowing you to purge it without recycling the server. The SEGMENT keyword causes the spin to happen after the specified number of pages have been written. Pages are defined by form-feeds sent to the DD. In V6.1 we provided a way to cause form feeds to be written at regular intervals enabling you to spin output at regular time intervals (a form-feed every hour with SEGMENT=24 spins the output once a day). In V7 we enhanced this support to let you configure WAS to send a form-feed after a specified amount of output. This lets you spin the output after a configurable amount of stuff was written to the DD as an alternative to just regular time intervals.Now in V8 we are adding another choice. You can still configure the server to write form feeds at intervals or based on volume of output. But in addition you can just force a form feed to be written whenever you like. This could be driven by your own automation or by an operator in response to output accumulating faster than you like. To trigger a form feed just use the new MODIFY command FORMFEED. Like this:

MODIFY server,FORMFEEDThis command will send a form feed to both the SYSPRINT and SYSOUT DD cards. Alternatively, if you are using the by-volume controls for form feeds you can use this command to change the volume limit being used to trigger form feeds. For example:

MODIFY server,FORMFEED,STDOUT=2000

This will change the current value for ras_stdout_ff_line_interval to 2000. You can also specify STDERR to change ras_stderr_ff_line_interval. This values will be in effect until the servant region restarts (newly started servants read the environment variable, not the dynamic value, just like other dynamic trace configuration changes). Note that STDOUT goes to the SYSPRINT DD card and STDERR goes to the SYSOUT DD card.

WLM AE_SPREADMIN Option

This one is a little tricky. Basically what we have done is take a hard-coded parameter that WebSphere passes to the z/OS Workload Manager (WLM) and made it configurable. So, in short, you can set the new wlm_ae_spreadmin WebSphere variable to either '0' or '1'. It defaults to '1' which means that WAS will continue to specify AE_SPREADMIN(YES) on its call to the WLM IWM4SLI interface. If you specify a zero, that causes WAS to specify AE_SPREADMIN(NO).And that's all there is to it. Yeah, ok. So what does this do for you? We should start by quoting the description of this parameter from the WLM book:



AE_SPREADMIN=NO The server tasks specified in AE_SERVERMIN will be distributed to service classes as needed in order to meet goals.

AE_SPREADMIN=YES The server tasks specified in AE_SERVERMIN will be distributed as evenly as possible to all service classes being used to execute work requests

So what does that mean? We should consider an example. Suppose you have a server configured with four servant regions. Suppose the work received by this server gets classified into two different service classes. In previous releases WAS hard-coded AE_SPREADMIN=YES. So WLM will divide the four servants up evenly between the two service classes. So each service class gets two servant regions. WLM might only use one if there isn't enough workload in the service class to use both. Furthermore, if one of the service classes can't meet its goal with two servant regions, WLM is forced to start a fifth servant (if the configuration allows) to assign to that service class. It won't change the service class a servant region is bound to in order to meet goals if that creates an uneven balance of servant regions assigned to each service class.But suppose that you have WAS V8 installed and you set wlm_ae_spreadmin to '0'. That causes WAS to tell WLM AE_SPREADMIN=NO. Now WLM will balance assignment of servant regions to service classes as it sees fit in order to help meet its goals. That might include changing the binding of an existing servant region to a different service class. In our example, this might lead to one service class with three servant regions and the other service class with only one. In fact, WLM will always keep at least one servant region for each service class, but it might decide to leave some servants unbound. Again in our example this could result in one service class with one servant region, the other service class with two, and one left unbound. The unbound servant stays up (because of the configured minimum number of servants) and is ready to be assigned to either service class as needed.

Displaying Server State

You can direct the modify command DISPLAY,SERVERS to any WAS on z/OS server. That server will report all the servers that are up in his cell, across the sysplex. For each server we will tell you the server's cluster and server name. We also tell you which system in the sysplex it is on, the ASID of the controller, and the service level of each server. Another modify command, PAUSELISTENERS, will cause the target server to close its listener ports and stop taking new requests (read about this in WP101138 – V6.1 Hidden Gems). If a server is up, but its ports are closed by PAUSELISTENERS, it would be nice if the display servers command would tell you that.In Version 8 the DISPLAY,SERVERS modify command has been enhanced to also report the 'state' of the server. There are four possibilities: ACTIVE, ENDING, PAUSED/STOPPING, and RECOVERY. ACTIVE seems pretty obvious. Basically ACTIVE means it isn't any of the other states; it could be up or it could be initializing. ENDING means that the server is on its way down. PAUSED/STOPPING means either you have issued PAUSELISTENERS or STOPped the server. It is kind of the same thing. In both cases the server is not taking new work, but there is a possibility work is still in-flight inside the server. The only difference is if we are stopping, then once the work completes the server will end. Finally, RECOVERY means that the server has been started to recover in-flight transactions and will automatically shut down once that is done. No new work will be taken.



Here's some sample output (adjusted slightly to fit the page):

BBOO0182I SERVER ASID SYSTEM LEVEL STATE BBOO0183I WAS00 /ZWASAXXX 6Fx SY1 8.0.0.0 (ff1106.32) ACTIVE BBOO0183I BBON001 /BBON001 58x SY1 8.0.0.0 (ff1106.32) ACTIVE BBOO0183I BBOC001 /BBOS001 5Bx SY1 8.0.0.0 (ff1106.32) PAUSED/STOPPING BBOO0183I BBODMGR /BBODMGR 57x SY1 8.0.0.0 (ff1106.32) ACTIVE

WLM Delay Monitoring Miscellaneous StatesFor quite a few releases WebSphere has used WLM Performance Blocks to enable delay monitoring. What WebSphere does, seriously over-simplified, is acquire a WLM Performance Block for each piece of work. The Performance Block (or PB) gets assigned a 'state'. WLM samples the states of the performance blocks and provides the results via SMF records. RMF reads those records and produces a report showing what states it found the PBs in what percentage of the time. You can read about all of this in the WebSphere Information Center here:http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.zseries.doc/info/zseries/ae/rprf_wlmdm.htmlThe tricky bit to this is that WLM has pre-defined the states a piece of work can be in. They tried to pick some likely sounding names like 'ACTIVE' and 'ACTIV_APPL' (meaning running in the application). For waiting states (what delay monitoring is all about) they tried to guess reasons you might be delayed. They came up with things like WAITING_DISTRIB which means we are waiting for something off-platform to respond to us. But the WLM developers realized there would probably be delays that didn't fit nicely into their pre-defined categories. So they added some 'miscellaneous' states numbered 1 through 15. If you have a delay in a miscellaneous state, you have to look up in the product documentation to find out what the product used that state for. Of course it can vary from product to product because each subsystem that uses PBs to report on delays can use the miscellaneous states for different purposes.After a while we realized that having to go look these up every time was a bit tedious. If we could just provide a clue in the RMF report it might be enough to nudge people's memory and save them a trip to the InfoCenter. As a result, RMF agreed to provide a terse explanation of the miscellaneous fields if WLM could give them the data in the SMF records. WLM agreed to provide the data in the SMF Type 72 Subtype 3 records if the users of the Performance Block APIs would explain what they were using the fields for. And WebSphere agreed to call the new API (IWM4MGDD) to provide our explanations for the miscellaneous fields.And thus you should begin to see RMF delay monitoring reports with a little table at the bottom that shows a short explanation for the miscellaneous delay states. For example, Miscellaneous delay state '4' is 'RRS' meaning that we are waiting for RRS to do something.Hopefully this will make reading these RMF reports a little easier.

Improved Message RoutingMessages issued by WAS (or by applications running inside of WAS) can go a few different places. Some messages go to the 'error log' which might be directed to the SYSOUT DD card. SYSOUT can be directed to the JES Spool or to a file. The file might be a dataset or a file in the USS file system. Or the 'error log' can be directed to a z/OS logstream. All that counts as one destination to WAS. Its just the error log.



A message can also be issued as a WTO (Write To Operator). WAS has some choices to make with a WTO because the route code used when the WTO is issued determines whether the message goes to the job log or to the actual console (sometimes called 'to glass'). In this case a parameter on the call to WTO determines where the message goes. But how does WAS decide whether to write to the error log or a WTO? And if its a WTO, what route code value should be used? What about application issued messages?For messages issued inside of WAS it depends on how the code was written that issues the message. When we decide to issue a message we decide where it should go and write the code appropriately. There is not (or at least was not previously) a way for you to influence this through configuration. For application issued messages it is much the same. Your application has to decide where the message should go and use the proper API to get the message sent there.Do you have any control? The only control that really works is a part of z/OS called MPF (Message Processing Facility). MPF allows you to change the attributes of specified WTOs. So basically if we thought a message should be a WTO to the console, you can change it to just go to the joblog/hardcopy. That's about it though.Until Version 8! In Version 8 we have introduced four new environment variables that relate to messages. Each of these variables corresponds to a message destination. The value you specify for the variable defines the messages you want to be sent to that destination, regardless of what the code that issued the message tried to do.For example, if you defined the value of ras_message_routing_errorlog to be BBOO0001I then the “Control Region Starting” message would go to the error log instead of to the console. That's probably an odd use of the facility though. A better use would be to find a message that normally goes to the error log that you would like to use as a trigger for automation. The problem is that automation does not see messages written to the error log, only WTOs. So suppose message BBOO9999E (I made that one up) goes to the error log, but you want to write some automation to take action when this message is issued. You can set ras_message_routing_hardcopy=BBOO9999E and now this message will be a WTO to the joblog/hardcopy.The other two variables are ras_message_routing_console and ras_message_routing_none. The first of these two will cause the specified messages to be written as WTOs to the console. This makes a message that was previously written to hardcopy or the error log visible to an operator. The second one ('none') tells us to suppress the message. It will not be written anywhere.A word of warning about turning error log messages into WTOs. Some customers, especially those in Japan, will use the z/OS MMS facility (MMS stands for MVS Messaging Service) to use skeletons to translate WTOs from English into Japanese. IBM provides Japanese skeletons for those messages we know are written to the console. If you use this new capability to route messages to the console that do not normally go there, there will be no MMS skeleton to match it and the message will just come out in English. And what about those messages that get 'wrapped' by the BBOO0222I message. WAS on z/OS wraps messages whose message ID does not comply with z/OS message standards with this special message ID to stay within the rules. The 'real' message ID follows the BBOO0222I message identifier. The message routing code recognizes the special '222' messages and looks at the inner 'real' message ID to decide if it should be routed somewhere else. Therefore if you want to alter the routing for one of these messages put the 'real' message ID in the setting for the appropriate message routing environment variable.What about dynamic capability? Wouldn't it be cool to be able to dynamically change where messages are routed? Suppose some piece of WAS code goes nuts and suddenly starts throwing



hundreds of messages to the console. Suppose it is our made up message BBOO9999E. Then you could issue:MODIFY server,MSGROUTE,NONE=BBOO9999EThis will cause WAS to suppress that message ID. You can also specify ERRORLOG, HARDCOPY, and CONSOLE on the modify command. This lets you dynamically change what messages are being routed. If you change your mind about a dynamic change you can specify RESET. That tells WAS to go back to the original environment variable setting for routing messages to this destination. For example:MODIFY server,MSGROUTE,NONE,RESETIf you just want to stop whatever you've done to WAS message routing and go back to the way the code was written, use the Modify CLEAR option, like this:MODIFY server,MSGROUTE,NONE,CLEAROf course the RESET and CLEAR options are available on all four message destinations.If you have lost track of the changes you have made then you can use the new display support to find out what the current settings are. Just issue this:MODIFY server,DISPLAY,MSGROUTEThis command will show you the current settings for all four destinations. Or you can ask for just one like this:MODIFY server,DISPLAY,MSGROUTE,CONSOLEOr any of the four destinations. Of course you can specify more than one message to influence. Just separate the message IDs with commas like this:ras_message_routing_hardcopy=BBOO9991E,BBOO9992E,BBOO9993E

There is a lot of potential in these options to get rid of messages that annoy you and to give more capability to your automation by making visible messages it previously couldn't see. But be careful. Like Uncle Ben says, “With great power, comes great responsibility”. (That's the Uncle Ben from Spider Man, not the one who makes rice.)

Eliminate Redundant Tracing of Logged MessagesWhile we are talking about where messages go, we should talk about tracing of messages. We try to put tracing into the code wherever anything interesting is going on. That helps us debug any problems that might turn up. If we are about to log a message, that is pretty interesting. That is why we put in a trace point to trace the message we are about to log – in case something went wrong in the logging code we would have a trace of the message that caused the problem.Well that seems reasonable, until you notice that the result is that every message we write turns up in the job output twice – once in SYSOUT and once in SYSPRINT. The first is the actual message and the second is the trace of the writing of the message.More than one customer has complained about this apparently pointless redundancy. We always defended our decision by pointing out that it would be difficult to debug a problem in the logging code if we had no tracing to tell us we were logging something.



In Version 8 we tried to find some middle ground on this. To understand what we did you have to remember a little about how tracing from the native WAS code works. In the native code there are three levels of tracing: Exception, Basic, and Detailed. Exception tracing was intended to be used to trace any unusual event. Basic tracing was for trace points that were essential to the flow of a request through the server. Detailed was, obviously, for more detailed information. Knowing all that, what classification should we use to trace the fact that we are about to write a message to the log? Before you answer, remember that traditionally the log (usually directed to the SYSOUT DD card, but able to be redirected to a logstream) is called the 'error log'. From the z/OS specific native code we are usually pretty sure that a message being written to the error log is actually an error. However, a message coming from the Java code is probably coming from code written for all platforms. WAS on distributed does not have this error log concept. Long ago when we began to merge the z/OS and distributed implementations of WAS we had to make some decisions about these sorts of situations. We decided at the time that any message that was created in Java, with a few exceptions, was an 'error' and thus should go to the error log.Sometimes that is true. Sometimes it is not. But, if we are writing a message to the error log, it seems reasonable that the trace used to record that would be an exception trace. After all, an error should be an exception, right?This is where we get in trouble. Most customers enable Exception tracing but do not turn on Basic or Detailed tracing. And because the trace that we are writing to the error log is an exception trace you normally get a trace for every message we log.For Version 8 we changed our minds. Any message that comes from the WebSphere common code that is routed to the error log is not necessarily an error. The trace of the fact that we are logging such a message to the error log is now a Basic trace and not an Exception trace.A very long explanation for what is essentially a small change. Hopefully this will substantially cut down on the number of apparently redundant traces you see for messages written to the error log.

SMF 120-9 Affinity InformationIn Version 8 we expanded a little on the information we provided with a single bit in the SMF 120-9 record (for more on the 120-9 record see WP101342 or the InfoCenter). Which bit are we talking about? The field is called SM1209DZ. We set that bit in the 120-9 record when a request was routed to a particular servant region because of an affinity.What does that mean? Well, suppose an HTTP request comes into the server. This is a new guy; we've never seen him before. So his request is allowed to run in any servant that is part of this server. And it does. As the request executes it creates an HTTPSession object. This is something like a shopping cart for this user. It is an object in the memory of this servant region. When another request from the same user comes in the application that runs is going to want to find this data to remember what it was doing. For example, if he clicks 'buy' you want to know what stuff he decided to buy. Customers hate it when you just guess. Therefore, when his 'buy' request comes in we have to be sure that the request runs in the same servant region that the original request ran in and created the session object. To make that work we tuck something special in the response we send back to the original request. On subsequent requests the client (the browser probably) sends back that same special thing. We use that to find the right servant region. For those who like to dig under the covers, we do this with a z/OS GRS ENQ and the special thing we send back includes the ENQ RNAME.But we should get back to the SMF record. What does that SM1209DZ bit mean? We set that bit when we place a request in a servant region because it has affinity to that servant region. In our example that would mean that the 'buy' request would be queued with affinity. The first request that



created his shopping cart defined the affinity, but would not have the SM1209DZ bit set because it was allowed to run anywhere.This lets you look at your SMF data and see how many requests ran in each servant region because WLM decided to run it there and how many ran in each servant region because we had no choice. But what you can not do is tie the requests together. You can not tell that 'this' request that WLM was allowed to route created an affinity that was later used by 10 requests, or only 1, or never. To help answer these questions we added a few fields to the z/OS Request Information Section of the 120-9 record. The version number of this section (SM1209CL) was incremented to allow you to tell if the data is there or not.The first pair of fields we added are SM1209GH and SM1209GI. These fields are set to the length and value of the special string we used for the ENQ if this request created an affinity. Thus a non-zero value in SM1209GH tells you that this request created an affinity.The next pair of added fields are SM1209GJ and SM1209GK. If the SM1209DZ (affinity) bit is set, then these two may be set to indicate what string was used to find the right servant region. It is possible to have SM1209DZ set and not have an affinity string. There are cases, especially MBeans, where a request is deliberately routed to all the servant regions and thus the requests are queued with affinity to a servant but it is not because of a previous request. But in more normal cases you should be able to take the string in SM1209GK and match it to a string that appeared in a previous request record in SM1209GI. That will let you construct the chain of SMF 120-9 records for this affinity. You can then calculate things like the average number of subsequent requests that come from an established affinity. Affinities mess up WLMs ability to balance work between the servant regions. The more requests in an affinity-chain the less you are allowing WLM to control things. You might also be able to notice some requests that establish an affinity that no subsequent request uses. These might represent 'abandoned' shopping carts, which could be interesting to know about. Or possibly a problem in the application because it is unnecessarily creating an HTTPSession that is never used. Or it might be something else entirely....Hopefully these new piece of data about requests will let you find out something you did not know about your environment.

Optimized Load Modules

Finally, a hidden gem you don't have to do anything to take advantage of! Well, ok, maybe buy some new hardware, but that's it!

One of the cool things about Java is that the Just In Time compiler (JIT) can determine what hardware you have and optimize the generated code to take advantage of whatever capabilities your hardware has. So if you buy a new system that has new instructions the JIT will immediately start building code to exploit them. Of course this assumes you have a JVM and JIT recent enough to recognize the new hardware.

Native code (C, assembler, and such) doesn't have that advantage. IBM ships pre-compiled load modules that have to be built for the lowest common denominator platform. For WAS on z/OS that means the z800. There's a good chance you've got hardware that is more recent than that. And because the vast majority of the code in WAS (and probably the code in your WAS applications) is



written in Java, the JIT is taking advantage of the newer instructions available on your newer hardware. But what about the native code?

We could build separate copies of the native code, compiled for each hardware platform we support and make you install the right one. That is probably a little much for everybody. In Version 8 we decided to at least do something about this and try to make it easy for you too.

If you look in the 'lib' directory you will see some new directory entries underneath. Things like s390-common, s390-31, s390-64, and s390x9-64. The s390-common directory contains native load libraries that we need regardless what bit mode the server is running or what hardware you have. The '-31' directory is obviously 31 bit. The other two, ending in '-64' are for 64 bit. The runtime code examines the configuration and adjusts the path used to enable us to get the code for the right bit mode. But why are there two for 64 bit? Ah ha! Because one of them contains DLLs that are optimized for z196 hardware!

As part of deciding if we need the 31 or 64 bit DLLs we also take a peek at the hardware you have and, if you are in 64 bit, will pick the best load modules for your hardware. And it happens automatically.

You might be wondering why we just shipped optimized DLLs for 64 bit and not 31 bit? Remember that in Version 7 we deprecated 31 bit. We are trying to move away from 31 bit servers and this sort of optimization only being done for 64 bit is just a piece of that.

You might also be wondering what the 'x9' stands for. Seems like x9 should be for the z9 and not the z196, but then wouldn't it be z9 and not x9? So confusing. Actually not. The 'x' stands (more or less) for 'architecture' and the '9' is the architecture level we used when compiling the code. Look in the C++ User's Guide for the ARCHITECTURE compiler option.

But the real question is, how much does this help? Actually at this time I don't really know. Given that the vast majority of the path to dispatch a request is in Java, it will not be a huge help to performance. But any improvement is an improvement. And this is just one more reason to get that new machine!

Tagdata in the 'ps' and Display,OMVS,PID CommandIt was brought to our attention that the Unix System Services process-status command (ps) can display tagdata for the process. The tagdata is set by the process calling a USS API. We thought that was pretty cool, and thus we added some tag data for the WAS processes. The tag takes the following form:WAS: <cell>/<node>/<cluster>/<server> - <space type> - <service level>So for example:

WAS: WAS00/NDN1/BBOC001/BBOS001 - CTL-AS - HH110309Thus this tag is from a control region. The '-AS' indicates it is an application server. Other options include '-NA' for Node Agent and '-DM' for Deployment Manager. This control region is in the WAS00 cell, in the NDN1 node, in the BBOC001 cluster. The server name is BBOS001. The maintenance level is HH110309. Here's an example of the output from the command, complete with options and a 'grep' to filter out other non-WAS processes:



/u/MSTONE1:>ps -m -Ao jobname,pid,stime,etime,thdcnt,args,tagdata | grep -E "BBO|WAS:"

BBOS001 83952584 11:53:24 00:04:27 1 BPXBATA2 -BBODMNC 33620956 11:53:24 00:04:27 1 BPXBATA2 -BBODMGR 83952605 11:53:29 00:04:22 1 BPXBATA2 -BBOS001 50398175 11:53:24 00:04:27 90 /WebSphere/ND/AppSer -- - - - - - WAS: WAS00/NDN1/BBOC001/BBOS001 - CTL-AS - HH110309BBODMNC 33620971 11:53:24 00:04:27 30 /WebSphere/ND/Deploy -- - - - - - WAS: WAS00/NDN1/WAS00/ZWASAXXX - DAEMON - HH110309BBON001 67175410 11:53:45 00:04:06 1 BPXBATA2 -MSTONE19 66554 11:57:51 00:00:00 1 grep -E BBO|WAS: -BBODMGR 33620996 11:53:29 00:04:22 86 /WebSphere/ND/Deploy -- - - - - - WAS: WAS00/WAS00/BBODMGR/BBODMGR - CTL-DM - HH110309BBOS001S 16843787 11:54:37 00:03:14 1 BPXBATSL -BBOS001S 66584 11:54:37 00:03:14 39 /WebSphere/ND/AppSer -- - - - - - WAS: WAS00/NDN1/BBOC001/BBOS001 - SR - HH110309BBON001 33621019 11:53:45 00:04:06 88 /WebSphere/ND/AppSer -- - - - - - WAS: WAS00/NDN1/BBON001/BBON001 - CTL-NA - HH110309BBODMGRS 16843807 11:57:17 00:00:34 1 BPXBATSL -BBODMGRS 66594 11:57:17 00:00:34 18 /WebSphere/ND/Deploy -- - - - - - WAS: WAS00/WAS00/BBODMGR/BBODMGR - SR - HH110309

You can also use the MVS console DISPLAY,OMVS command specifying the process id (PID) of a WAS process and see this same information. Here's an example that shows a Daemon:D OMVS,PID=65767 BPXO040I 15.24.24 DISPLAY OMVS 079 OMVS 000E ACTIVE OMVS=(KB) USER JOBNAME ASID PID PPID STATE START CT_SECSASCR1 BBODMNC 00B1 65767 50397223 HR---- 19.30.04 .333 LATCHWAITPID= 0 CMD=/WebSphere/ND/DeploymentManager/lib/bbod THREAD_ID TCB@ PRI_JOB USERNAME ACC_TIME SC STATE 3143E00000000000 006E6E00 .121 SPM YU TAG=WAS: WAS00/NDN1/WAS00/ZWASAXXX - DAEMON - EE110219 3143F00000000001 006E6A00 .001 PTX JR V 3144000000000002 006D4CF0 .001 PTX JR V 3144100000000003 006E6868 .001 PTX JR V 3144200000000004 006E6648 .001 PTX JR V 3144300000000005 006E6428 .001 PTX JR V 3144400000000006 006E6208 .001 PTX JR V

With a little careful use of 'grep' you can probably build little shell scripts to quickly show you interesting things about the servers that are up. One usage that was already suggested was to quickly hunt for any servers that are up on the current system that are using a particular service level (just grep the 'ps' output for the service level you care about).



Thread Hang Recovery Diagnostic Improvements

You might remember that in Version 7 we introduced some changes in how dispatch timeouts are handled. If you don't remember, then you might want to take a look at WP101374. Or just keep reading and I'll try to hit the main points. Prior to Version 7 a timer was started in the controller when a request was put on the queue to be dispatched. If the timer expired while the request was in dispatch on a thread in a servant region the controller would abend the servant region. In Version 7 we allowed the dispatch thread to register something called an ODI. If the dispatch timer expired and there was a registered ODI, we would call it (on another thread in the servant region) and hopefully the ODI could break the dispatch thread out of whatever it was that was taking so long. As an example, the DB2 Type-2 JDBC connector implemented an ODI that can try to cancel your DB2 request if we timeout while your application request dispatch is in DB2. We also implemented, internal to WAS, a “JVM ODI” that uses Java services to interrupt a dispatch thread that is hung in a Java wait (such as a socket.read( )). This helps us break out of hangs waiting for Type-4 connectors. The point of all of this was to allow us to still avoid hangs and deadlocks through timeouts but without having to abend and restart an entire servant region. For the most part, as far as we have heard anyway, we have been pretty successful helping customers avoid timeout abends while still having the capability of timing out requests. However (there's always a 'however') we have gotten some feedback that, even though we avoided the abend, it is difficult to figure out what exactly was going on that caused the hang. In Version 8 we have tried to improve things. Before we get into the new things in Version 8, we should go through the sequence of events for a timeout as it works in Version 7.

1) Dispatch begins2) The dispatch timer expires3) WAS issues message BBOJ0113I (see examples below)4) If there are any ODIs registered (there's always at least one) it gets called to interrupt the

dispatch (if possible)5) Wait a bit and repeat #4 until we run out of ODIs or just give up6) Once we've given up we collect the configured documentation (e.g. callstack of the dispatch

thread) and notify the controller that the thread is hung7) The controller will, since we gave up, abend the servant (mostly..this gets complicated, go read

the other timeout papers for details). The controller issues the BBOO0327I message.Of course if the dispatched request responds to being prodded by an ODI it might complete on its own. In that case we skip steps 6 and 7, but the controller will still issue BBOO0327I to let you know the request did not complete without help.In Version 8 we have added a few things to this process. Perhaps the most important is the option to collect documentation BEFORE we meddle with the dispatch of the request. In Version 7 we do not collect documentation until step 6, by which time we have possibly interfered with the request several times. It might be nice to know what the request was originally stuck doing in addition to knowing what it was doing when we gave up trying to get it to complete. To enable the server to gather documentation prior to interrupting the request, set a servant JVM property called com.ibm.ws390.interrupt.applyDumpActionPreInterrupt to true. That will cause us to gather the



same documentation (e.g. callstack) that we will gather when we give up before we start meddling with the dispatch of the request. The rest of the changes involve issuing more messages in the servant region to let you know what is going on with the request. The first new message is BBOJ0123I. This message is issued before we start meddling with the request. The intent of the message is to provide you with a lot of information about the request that has timed out. We will see some examples below.The next new message in Version 8 timeout processing is BBOJ0122I. As part of implementing an ODI you have to provide a method WAS can call to get information about your ODI. This is mostly to support the DISPLAY,THREADS command. But it might be nice, as we are about to call the ODI to interrupt the dispatch, to let the ODI tell us what it thinks is going on that it will try to interrupt. So BBOJ0122I basically shows you the results of asking the ODI we are about to call what it thinks is going on. It is possible we will drive multiple ODIs for a request as we try to get it to complete. We will issue BBOJ0122I for every ODI we call. It is possible that you might have a lot of timeouts and the registered ODIs are able to interrupt the requests quite nicely and keep everything running smoothly. If this is all ok, you might not want the BBOJ0122I messages. If you'd like to disable them, just set com.ibm.ws390.interrupt.disableBBOJ0122I to true (that is, it is true that you would like to disable the messages). Finally, when we give up we will issue another new message, BBOJ0124I, to let you know we gave up and provide information about the request. The information in this request is very similar to the information in BBOJ0123I, but comes at a different point in time (after we've given up meddling with the request). I should point out that we have similar changes in the CPU timeout processing. There are no ODIs for CPU timeout, but we do still issue some similar messages. To keep the timeout types separate we have different message IDs. We use BBOJ0114I instead of BBOJ0113I and we use BBOJ0125I instead of BBOJ0124I. CPU timeouts do not issue a BBOJ0122I or BBOJ0123I type message.Now that we know all this, we should go back to our timeout sequence of events and update it for Version 8. Then we will take a look at some examples of the new messages and see what is in them.

1. Dispatch begins2. The dispatch timer expires3. WAS issues message BBOJ0113I (see examples below)4. WAS issues BBOJ0123I with details about the request5. If configured to applyDumpActionPreInterrupt gather the configured documentation (e.g.

callstack)6. If BBOJ0122I is not disabled, call the ODI to get information about it and issue the message7. If there are any ODIs registered (there's always at least one) it gets called to interrupt the

dispatch (if possible)8. Wait a bit and repeat #6 and #7 until we run out of ODIs or just give up9. Issue message BBOJ0124I with detailed information about the request and to indicate we

gave up10. Once we've given up we collect the configured documentation (e.g. callstack of the dispatch

thread) and notify the controller that the thread is hung



11. The controller will, since we gave up, abend the servant (mostly..this gets complicated, go read the other timeout papers for details). The controller issues the BBOO0327I message.

Again, the BBOO0327I message will be issued by the controller whether we gave up or not.Now we should take a look at some examples of the messages. Here is BBOJ0113I which is issued when we first start to meddle with any dispatched request. The number at the end is just a correlator for this request to help you match this message with other messages related to the same request. This message marks the beginning of our meddling with the request. If you see this message you should always see a matching BBOO0327I message from the controller. Other messages depend on what happens. Here's the example:

BBOJ0113I: The Interruptible Thread Infrastructure is attempting to advance work running under request ffff18b2

Next up in is BBOJ0123I. This message provides more details about the timed out request and is issued before we start to meddle. Here's an example:

BBOJ0123I: The Interruptible Thread Infrastructure is attempting to advance work running under request ffff18b2, request details: ThreadDetails: ASID = 0129, TCB = 0X006C62D8, Request = ffff18b2, Is JVM Blocked = false, Tried to interrupt = false, Given up = false, Internal Work Thread = false, Hung Reason = Not Hung, SR Dispatch Time = 2011/02/25 20:36:56.474373, CTL Receive Time = 2011/02/25 20:36:56.352540, CTL Queued to WLM Time = 2011/02/25 20:36:56.471058, Request Timeout limit = 63, Elapsed Execution Time = 65, CPU Time Used Limit = 3500000, Outbound Request Timeout Limit = 30, ODI Details = [JVM INTERRUPTIBLE THREAD, Monitor ACTIVE]

So what is all that? Well right there at the end of the first sentence is that request identifier again. It matches the same value (ffff18b2) we saw in the BBOJ0113I message. After that we get more detailed information. There is the servant region ASID (0129), the address of the dispatch TCB (6C62D8), and the request identifier (again..). Then we ask the JVM ODI what it knows. It tells us, in this case, that we are not blocked in a Java wait. We have not yet tried to interrupt the request (but we are about to), and, of course, we have not yet given up. This is an application request, so it is not running on an internal work thread. Those threads are used for requests from the controller to the servant and are also subject to timeouts. If this part of the message indicates 'true' then this is probably an IBM internal thing (although it might be an application-installed Mbean). Next up is the 'hung reason'. At this point it is marked as 'not hung' because we have not yet given up trying to get the request to move along. As we will see later this field can indicate the dispatch timer has run out, in other words we gave up, or that the CPU timer has run out. Following the hung reason we have three time stamps to help you determine how this request fits into the flow of activity on your system. The last few fields are some specific time values. We have the actual dispatch timeout used for this request (63 in this case) and the amount of time actually elapsed (65 seconds in this case). We also tell you the CPU timeout (in milliseconds) and the timeout value that will be used if this request makes an outbound IIOP request (30 in this case). Finally, we call the registered ODIs and ask them for any information they can provide about the request and what it is doing.The next message of interest is the new BBOJ0122I message. This message is issued just before we call an ODI to interrupt a request and allows the ODI to tell us what it thinks about what is going on with the request. Here is an example from the JVM ODI, which does not know very much so it



just tells us that it is an 'active' ODI. Other more interesting ODIs will hopefully provide more interesting information.

BBOJ0122I: The Interruptible Thread Infrastructure about to drive a ODI to advance work running under request ffff18b2, ODI details: Monitor ACTIVE

The last message to look at in detail is the BBOJ0124I message which is issued when we give up trying to get a timed out request to complete. Here is an example:

BBOJ0124I: The Interruptible Thread Infrastructure timed out a request and it has become unresponsive, request ffff18b2, request details: ThreadDetails: ASID = 0129, TCB = 0X006C62D8, Request = ffff18b2, Is JVM Blocked = false, Tried to interrupt = true, Given up = true, Internal Work Thread = false, Hung Reason = Dispatch Timer Popped, SR Dispatch Time = 2011/02/25 20:36:56.474373, CTL Receive Time = 2011/02/25 20:36:56.352540, CTL Queued to WLM Time = 2011/02/25 20:36:56.471058, Request Timeout limit = 63, Elapsed Execution Time = 65, CPU Time Used Limit = 3500000, Outbound Request Timeout Limit = 30, ODI Details = [JVM INTERRUPTIBLE THREAD, Monitor ACTIVE]

You can see this looks a lot like the BBOJ0123I message. Some differences to notice are the Hung Reason which is now “Dispatch Timer Popped”. Also note that we have indicated that “Tried to interrupt” is true and “Given up” is also true. As we noted earlier, for a CPU timeout different messages are issued but the contents are very similar. Here are examples of the BBOJ0114I and BBOJ0125I messages:

BBOJ0114I: The Interruptible Thread Infrastructure is attempting to interrupt a request ffff184e that exceeded its CPU Time

BBOJ0125I: The Interruptible Thread Infrastructure detected a request ffff184e that exceeded its CPU Time, request details: ThreadDetails: ASID = 012A, TCB = 0X006C64F8, Request = ffff184e, Is JVM Blocked = false, Tried to interrupt = false, Given up = true, Internal Work Thread = false, Hung Reason = CPU Time Exceeded, SR Dispatch Time = 2011/02/25 21:14:35.927278, CTL Receive Time = 2011/02/25 21:14:35.899603, CTL Queued to WLM Time = 2011/02/25 21:14:35.925408, Request Timeout limit = 63, Elapsed Execution Time = 0, CPU Time Used Limit = 4000, Outbound Request Timeout Limit = 30, ODI Details = [JVM INTERRUPTIBLE THREAD, Monitor ACTIVE]

Note in the BBOJ0125I message that Hung Reason is now “CPU Time Exceeded”. One more quick thing to point out and we will be done with this topic. If you ask for documentation to be gathered both before we interrupt the request and after we give up, how do you tell the difference and how do you relate it to all these other messages? Suppose the configured documentation to gather is 'CALLSTACK'. In that case you will get these two messages, one before we call the first ODI and, if we give up, the second after we have given up. Note the items in bold. The message text tells you why we gathered this documentation and the TCB address will match the TCB address in the other messages.

BBOJ0117I: JAVA THREAD STACK TRACEBACK FOR THREAD WebSphere WLM Dispatch Thread t=006c62d8: Hung Thread Recovery--pre-interrupt Traceback for thread WebSphere WLM Dispatch Thread t=006c62d8: com.ibm.ejs.ras.CB390TraceEventListener.writeTrace(Native Method) com.ibm.ejs.ras.CB390TraceEventListener.processEvent(CB390TraceEventListener.java:390) com.ibm.ws.logging.WsHandlerWrapper.publish(WsHandlerWrapper.java:43) java.util.logging.Logger.log(Logger.java:1121) com.ibm.ejs.ras.Tr.logToJSR47Logger(Tr.java:1714) com.ibm.ejs.ras.Tr.fireEvent(Tr.java:1672)



com.ibm.ejs.ras.Tr.fireTraceEvent(Tr.java:1591) com.ibm.ejs.ras.Tr.entry(Tr.java:845) com.ibm.ws390.interrupt.RoadKillCppUtilities.deregisterRequest(RoadKillCppUtilities.java:109) com.ibm.ws390.orb.DispatchInterceptor.endDispatch(DispatchInterceptor.java:257) com.ibm.ws390.orb.ServerRegionBridge.httpinvoke(ServerRegionBridge.java:220) com.ibm.ws390.orb.CommonBridge.getAndProcessWork(CommonBridge.java:573) com.ibm.ws390.orb.CommonBridge.runApplicationThread(CommonBridge.java:509) com.ibm.ws.util.ThreadPool$ZOSWorker.run(ThreadPool.java:1836)

BBOJ0117I: JAVA THREAD STACK TRACEBACK FOR THREAD WebSphere WLM Dispatch Thread t=006c62d8: Thread Hang Recovery--thread could not be encouraged to complete Traceback for thread WebSphere WLM Dispatch Thread t=006c62d8: com.ibm.ejs.ras.CB390TraceEventListener.writeTrace(Native Method) com.ibm.ejs.ras.CB390TraceEventListener.processEvent(CB390TraceEventListener.java:390) com.ibm.ws.logging.WsHandlerWrapper.publish(WsHandlerWrapper.java:43) java.util.logging.Logger.log(Logger.java:1121) com.ibm.ejs.ras.Tr.logToJSR47Logger(Tr.java:1714) com.ibm.ejs.ras.Tr.fireEvent(Tr.java:1672) com.ibm.ejs.ras.Tr.fireTraceEvent(Tr.java:1591) com.ibm.ejs.ras.Tr.entry(Tr.java:845) com.ibm.ws390.interrupt.RoadKillCppUtilities.deregisterRequest(RoadKillCppUtilities.java:109) com.ibm.ws390.orb.DispatchInterceptor.endDispatch(DispatchInterceptor.java:257) com.ibm.ws390.orb.ServerRegionBridge.httpinvoke(ServerRegionBridge.java:220) com.ibm.ws390.orb.CommonBridge.getAndProcessWork(CommonBridge.java:573) com.ibm.ws390.orb.CommonBridge.runApplicationThread(CommonBridge.java:509) com.ibm.ws.util.ThreadPool$ZOSWorker.run(ThreadPool.java:1836)

Handling Work With Affinity During a Timeout Delay

When timeout processing decides it has to abend a servant region there might be other requests in dispatch on other threads in that servant that are completely innocent. For that reason you can configure a timeout delay (control_region_timeout_delay or use the Modify TIMEOUT_DELAY command). This defers the abending of the servant region for the specified number of seconds. During that delay the servant region threads will finish what they are doing and not take any new work. This allows the innocent to get out of the way. But there is a complication. What about requests that come in during the delay that have affinity to the servant region we are going to abend? Well, they just sit in the WLM queue. The servant region dispatch threads will not pick up new work, but the servant is still a valid target for the request. These requests will remain in the queue until their queue timeout expires or, more likely, until the timeout delay runs out and we abend the servant. The abending of the servant region will either cause the requests to be cleaned up (rejected) or requeued to be able to run in any servant region. This requeue of the request is essentially the same as what would happen to the request if it came in just after the servant region was abended. We can not find the servant region it has an affinity to, so the request is allowed to run in any servant. The state data that caused the affinity should be tucked away in DRS and able to be recreated in any servant.Well then, why do we let these requests sit in the queue waiting for the abend? Why not just queue them to run anywhere right away? Because the request actually has an affinity to an active (but



soon to die) servant region. Allowing it run in a different servant region when the original one is not yet dead would cause a lot of confusion and problems. But letting the request just sit in the queue is bad. Somewhere there is a client waiting for a response. Should we just wait for the timeout delay to expire and then requeue it when it is safe? What if you only have one servant region? In that case the request will have to wait for the timeout delay to expire AND for a new servant region to initialize. That might take a while. Perhaps it would be better to just reject the request.So in V8 we changed the code a little bit and that is what happens. Any request that comes in with an affinity to a servant region that is going to be abended when the timeout delay expires will just be rejected. If the client re-sends the request a little later, after the servant has abended, then the request can run in any servant and will be queued without affinity just like before. Note that any requests with affinity that are actually in the queue to run when the servant is marked for an abend will also be rejected. This should keep clients with requests with affinity to a dying servant region from hanging during the timeout delay. Remember that we have the timeout delay to avoid impacting requests that were not responsible for the timeout that led to the abend. This helps us not hurt newly arrived innocent requests too badly either.



Document change historyCheck the date in the footer of the document for the version of the document.10/03/11 Initial Version10/4/11 Added document number

End of WP101992


websphere application server for z/os v8 hidden …...hidden gems in the service stream but wait,...

Documents