operating a websphere process server...

Operating a WebSphere Process Serverenvironment, Part 3: Setup, configuration, andmaintenance of the WebSphere Process ServerBusiness Process Choreographer databaseSkill Level: Introductory

Jonas Grundler ([email protected])Software ArchitectIBM

Rolf Baeurle ([email protected])Software EngineerIBM

David J. Bonaccorsi ([email protected])Software ArchitectIBM

Andreas Fried ([email protected])Software EngineerIBM

Fred Orosco ([email protected])Software EngineerIBM

Michael Poirier ([email protected])Software EngineerIBM

16 Dec 2009

Updated Apr 2010

Setup, configuration, and maintenance of the WebSphere Process Server Business Process Choreographerdatabase

Trademarks

© Copyright IBM Corporation 2009, 2010. All rights reserved. of 29

mailto:[email protected]






http://www.ibm.com/developerworks/ibm/trademarks/

http://www.ibm.com/legal/copytrade.shtml

Part 3 of this multi-part series provides database system specific information for initialsetup and maintenance of the WebSphere® Process Server Business ProcessChoreographer database. Where appropriate, Business Process Choreographerapplication level information is provided. This enables database administrators todiscuss with WebSphere Process Server application developers and architectsdecisions related to the database configuration. The information in this article can beapplied to WebSphere Process Server V6.0 and later. Features are marked if theyare available in later releases only. Special considerations are contained for thefollowing database systems: DB2® for LUW, DB2 for z/OS®, and Oracle®.Note: The article has been updated with DB2 for z/OS information.

Introduction

The Business Process Choreographer database (BPEDB) stores businessprocesses, human tasks, and related artifacts. The BPEDB is mainly accessed in thefollowing situations:

• Process instance and human task navigation

• Human task list and process instance list queries using the BPC API

• Other Business Process Choreographer (BPC) API calls

The more business process instances and human tasks are stored in the BPEDB,the more important the database configuration and maintenance. Also, the morecomplex the scenarios are, the more steps may be required to achieve performanceand stability goals. This article provides sample database configuration scripts forthe BPEDB, as well as steps to take for continuous database operation.

The sample scripts and configuration parameters have been adjusted for thefollowing setup and scenario as shown in Figure 1.

Figure 1. WebSphere Process Server setup

developerWorks® ibm.com/developerWorks


Trademarks




• Loading up to 300,000 process instances with 10 activities, such ashuman tasks or asynchronous invoke activities.

• Running automatic long-running process instances with 10 asynchronousactivities with a throughput of up to 25 processes instances per second.

• Processing 50 human tasks per second, as part of a process instance.

• Running up to 10 complex human task list queries in parallel, using querytables, at sub-second response times.

Each section in this article describes a particular database operation, in the contextof the BPEDB. Supporting sample scripts are provided for some operations, as wellas database system specific information. The settings discussed in this article areimportant in the most common scenarios. Specific environments and applicationsmay require individual configurations.

The SQL scripts have been tested with database system administrator privileges onthe following database versions:

• DB2® V9.1 for z/OS®

• DB2 V9.1 for LUW®

• Oracle® 11g

The scripts are designed for and have been tested on WebSphere ApplicationServer Microsoft® Windows® and WebSphere Application Server for z/OS on thespecified database systems and versions. Customizations are necessary to run the

ibm.com/developerWorks developerWorks®


Trademarks




scripts on UNIX® based systems and on other operating systems. All scripts are runon the BPEDB database server.

DB2 for z/OS

For WebSphere Process Server use with DB2 V9 for z/OS, you can use twodeployment patterns:

• WebSphere Process Server distributed to DB2 V9 for z/OS

• WebSphere Process Server for z/OS to DB2 V9 for z/OS.

When using DB2 for z/OS as the BPC database, it is important to consider initialsetup goals of the BPEDB and also a plan for regular maintenance of the database.Usually for z/OS, these DB2 images can be quite large and centralized as asubsystem that is scrutinized for performance and availability.

Figure 2 highlights two patterns that are used in DB2 for z/OS and Process Serverdeployments:

• DB2 for z/OS and WebSphere Process Server installed on z/OS

• DB2 for z/OS and WebSphere Process Server installed on a distributedsystem.

When using Process Server for z/OS, consider local DB2 JDBC Type 2 Connectorsupport. For additional information about DB2 z/OS JDBC Type 2 and Type 4considerations see, DB2 9 for z/OS Technical Overview.

Figure 2. WebSphere Process Server and DB2 for z/OS setup



Trademarks


http://www.redbooks.ibm.com/abstracts/sg247330.html



More DB2 users demand access to DB2 data every hour of the day, every day of theyear. DB2 data sharing helps meet your service objective by improving availabilityduring both planned and unplanned outages. When DB2 for z/OS is used with datasharing on z/OS, the result is a very high availability subsystem that meets thedemands of enterprises and mission critical data. If one subsystem fails, users canaccess the data from another subsystem. Transaction managers are informed thatDB2 is down and can switch new user work to another DB2 subsystem in the group.For unplanned outages, the z/OS Automatic Restart Manager can automate restartand recovery. For more information about data sharing, see DB2 9.1 for z/OS datasharing.

Initial setup

After database installation, initial configuration parameters are adjusted to allow thedatabase to make full use of the provided hardware resources. The most importantsettings relate to:

• Database memory



Trademarks


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.dshare/db2z_dshare.htm




• Database files distribution

• Resource serialization

• Initial SQL optimization assumptions

Providing enough memory enables the database for efficient read and complexquery operations. The distribution of the database files across all available physicaldisks is necessary for high-throughput process instance navigation and otherwrite-intensive scenarios. There are two main types of database files:

• Data files

• Transaction logs

Data files store information about tables and indexes. Transaction logs storeinformation about running transactions that modify data files. Transaction logs canbe updated frequently when high insert or data modification processing occurs. Forbest throughput, place them on dedicated high-performance disks.

For improved scalability in DB2 for z/OS, ensure that an adequate number ofin-memory log buffers are assigned. For details, see DB2 9.1 for z/OS Active logdata set parameters: DSNTIPL.

Data file distribution is configured at BPEDB creation time. There are no scriptsprovided for this with this article. However, sample configuration parameters, whichare customized after database creation, are provided in theBPEDB_initialSetup_db-system.sql script (Download section of the article).

Following the initial creation of the BPEDB database objects, all the tables are emptyand may gradually accumulate data over time. The initial empty tables and absenceof valid table statistics challenge the SQL optimizer as it attempts to determine theoptimal access path for SQL processing. The SQL optimizer must makeassumptions about data statistics when determining the access path. These initialassumptions may be adequate, but over time they will become invalid and cancause performance problems. As the data volume increases, the optimal accesspath needs to evolve based on valid table statistics. To help in this transition fromempty tables to populated tables, you can use techniques to influence theassumptions used by the optimizer to access data. See Maintaining databasestatistics for more details.

DB2 for z/OS

DB2 for z/OS is an enterprise wide database server that has been optimized forlarge implementations with utmost care for availability and scalability. With DB2 forz/OS, the database image (subsystem) can be very large, highly available, andallows for the isolation of multiple databases for various lines of business in the



Trademarks


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.inst/db2z_dsntipl.htm




enterprise. Figure 3 provides information about DB2 for z/OS data structures anddata sharing. Also see DB2 9.1 for z/OS data structures.

The brief description in this article shows how the BPEDB tables fit into an overallview of DB2 for z/OS. Figure 3 shows how some DB2 for z/OS structures containothers. To some extent, the notion of "containment" provides a hierarchy ofstructures.

Figure 3. DB2 for z/OS subsystem

The most important considerations regarding the configuration of the BPEDB are:

• Table to table space assignment as demonstrated by the default scriptsfor the BPEDB.

• Table space to buffer pool assignment and sizing as demonstrated by thedefault scripts for the BPEDB.

• Index tuning as the application workload evolves.

DB2 LUW

The most important considerations when configuring the BPEDB are:

• Table spaces need to be distributed among available disks. This must bedone at BPEDB schema creation time.

• Transaction log files need to be placed on a dedicated high-performing



Trademarks


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.intro/db2z_db2datastructures.htm



disk.

• Lock timeout and deadlock check time configuration parameters must beset appropriately.

Oracle

Important considerations regarding the configuration of the BPEDB are:

• Table space is distributed among available disks. This must be done atBPEDB schema creation time.

• Redo log files are on a dedicated high-performing disk.

• Automatic memory management is enabled.

• The statistics level is set to TYPICAL.

• The maintenance window for automatic maintenance jobs is customized.

BPEDB database schema

This section provides information about the BPEDB database schema.

Access patterns and characteristics

SQL type: The BPEDB is accessed by dynamic SQL only. Referential integrityconstraints are not used. Triggers, procedures, or functions are not used.

Process instance and human task navigation: Processing of processes instancesand human tasks requires fetching information from the BPEDB, as well as updatingand inserting data. The SQL statements involved in these situations run against asingle table only in most cases. Typically, the data is accessed by using the primarykey and select statements run within 10 milliseconds of CPU time. For databasesystems Derby®, Oracle, and Informix®, parts of the work is synchronized betweentransactions using exclusive locks on table PROCESS_CONTEXT_T. Therefore, lockcontention and longer-running SQL statements are expected in some situations.

Human task list and process instance list queries issued by the BPC API:Queries are issued by client applications using the BPC API to present human taskor process instance lists to the user. In most cases, more than one publisheddatabase view (see Database views for Business Process Choreographer) isaccessed to collect the requested information. These SQL queries may requirespecial tuning for good performance. Response time of these queries exceeds 50ms(at no disk I/O) in general.

BPC API calls other than human task and process instance list queries: Client



Trademarks


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.websphere.bpc.620.doc/doc/bpc/r6bpc_dbviews.html



applications run BPC API calls to perform actions such as claiming a human task orretrieving the input message of a process instance. These calls have various accesspatterns, including locking of certain objects, retrieval of LOB data, or deletion of anumber of rows in the BPEDB database. Accordingly, the response time of therelated SQL statements varies, and is dependent on the overall workload on thedatabase.

Disclaimer: The tables described in this section are not part of thepublic WebSphere Process Server API. Table structures are subjectto change without notice. Direct access to database content issupported through the BPC predefined database views only.

Tables with potentially high cardinality are described in this section. These tablesmainly store process instance and human task data. Together with the cardinalitiesin your environment, which are provided by the scriptBPEDB_gatherCardinalities_db-system.sql, the description of the tablesallows better assess to the contents of the BPEDB.

Table 1 lists the abbreviations and conventions used in this section.

Table 1. Abbreviations and conventionsAbbreviations or conventions Description

pi Number of process instances in the system

ai Number of long-running activity instances perprocess instance

ti (Number of task instances + number ofstandalone human tasks) per process instance +ai

ei Number of escalation instances per processinstance

u Number of humans in the staff repository allowedto access process instances and human tasks

optional Specific feature used in process instances or byhuman tasks, or used by BPC APIs. RelatedLOBs can be empty (optional).

underlined_t Table accessed by complex queries run usingthe BPC Query API

0..1 Zero or one, optional

0..m Zero or more (typically 0..10), optional

1..m One or more (typically 1..10)

1..M One or much more (typically 1..100..1000)

0..f Between zero and one (for example 0..0,5)



Trademarks




The cardinalities of the BPEDB tables depend on the number of process instances inthe database (Table 2). Human tasks, escalations, variable data, and audit loginformation are application dependent. Note that the human tasks tables alsocontain administrative tasks, which are not optional. The number of indexes is takenfrom the WebSphere Process Server V7.0 BPEDB database schema. Significantchanges to previous versions are documented.

Hints:

• Data persisted in loops, for example while-loops, is not completelyremoved from the database when the loop is re-iterated.

• The amount of persisted data belonging to a long-running processinstance is impacted mainly by the:

• Number of long-running activities in a process instance, includinghuman tasks

• Number of transactions of a process instance

• Size of the process variables

• Standalone human tasks are not associated with a process instance.

Table 2. BPEDB tables with potentially high cardinalityTable and number ofindexes

Typical cardinality LOB columns (typicalrange or size in KB)

Description

Process instance data

ACTIVITY_INSTANCE_ATTR_B_TIndexes: 5

pi * 0..m DATA (1..100): binarycustom properties;optional

Activity customproperties

ACTIVITY_INSTANCE_B_TIndexes:

• 7 since 7.0

• 5 in6.0-6.2

pi * 1..M (=ai) UNHANDLED_EXCEPTION(5): exceptional casesonlyPENDING_REQUEST_DATA(1..100): exceptionalcases only

Activity instances;roughly one for eachinvoke activity, taskinstance, wait activity,etc.

CORRELATION_SET_INSTANCE_B_TIndexes: 3

pi * 0..1 DATA_LONG (5): usedonly if the correlationset information isgreater than 3KB,optional

Correlation data;correlation sets must beexplicitly modelled

CORRELATION_SET_PROPERTIES_B_TIndexes: 1

pi * 0..1 DATA (2): correlationproperties; optional

EVENT_HANDLER_INSTANCE_B_TIndexes: 3

pi * 0..m - Created if an eventhandler is modelled onthe application



Trademarks




EVENT_INSTANCE_B_TIndexes: 5

pi * 0..m MESSAGE ( 1..100):REPLY_CONTEXT (5):used when a reply mustbe sent back to thecaller, optional

Created for inboundoperations

FOR_EACH_INSTANCE_B_TIndexes: 2

pi * 0..m - For-each instances;must be explicitlymodelled

INVOKE_RESULT_B_TIndexes: 1

pi * ai * 0..f INVOKE_RESULT(1..100): optional

Used in cases whereservice invocations arenot transactional

INVOKE_RESULT2_B_TIndexes: 2

PARTNER_LINK_INSTANCE_B_TIndexes: 2

pi * 1..m ENDPOINT_REFERENCE(5):SERVICE_DEFINITION(5):both LOBs are usedonly if a non-staticbinding of services isused; optional

Partner links, whichrepresent connectionsto external services

PROCESS_CONTEXT_TIndexes: 1

pi REPLY_CONTEXT (5):optional, used if theprocess has a two-wayinterfaceSERVICE_CONTEXT(5): J2EE context of theprocess instance starter

The J2EE context ofthe process instancestarter

PROCESS_INSTANCE_ATTRIBUTE_TIndexes: 3

pi * 0..m DATA: modelled binaryprocess customproperty, optional

Process customproperties

PROCESS_INSTANCE_B_TIndexes: 10

pi UNHANDLED_EXCEPTION(5): exceptional casesonly

Process instanceinformation

PROGRESS_COUNTER_TIndexes: 1Since: 6.1

pi * 0..1 - Used for eventsequencing

RELEVANT_SCOPE_ATASK_TIndexes: 2Since: 6.1

pi * 0..m - Administrative settingsfor scope instances

REQUEST_INSTANCE_B_TIndexes: 2

pi * 0..m REPLY_CONTEXT (5):optional

Used forrequest/responseoperations with apending reply

RESTART_EVENT_B_TIndexes: 2

pi INPUT_MESSAGE (5):the process instanceinput messageREPLY_CTX (5): thereply context of the

Created for eachprocess instance; usedin restart processinstance scenarios



Trademarks




process instance,optional

SAVED_ENGINE_MESSAGE_B_TIndexes: 2Since: 6.1

0..M ENGINE_MESSAGE_L(5):the navigation messageto process

Used for processinstance navigation

SCHED_TASKIndexes: 3

pi * 0..m +pi * ti * 0..f

TASKINFO (5): taskinformation

Scheduled tasks suchas scheduled deletionof process instances orwait activities

SCOPE_INSTANCE_B_TIndexes: 7

pi * 1..m BPEL_EXCEPTION (5):exceptional cases only

Scope instances

QUERYABLE_VARIABLE_INSTANCE_TIndexes: 10

pi * 0..m - Query properties

Human task data

REPLY_HANDLER_TIndexes: 2

pi * 0..m REPLY_HANDLER(1..100): optional

Used when a replyhandler is set onhuman tasks

RESULT_AGGREGATION_INSTANCE_TIndexes: 3Since: 7.0

pi AGGR_LOCATION (1):XPath expression ;application dependentAGGR_CONDITION (1):condition; optionalPREFIX_MAP (5): prefixto namespacemapping; applicationdependent

Used in parallel routingscenarios

TASK_HISTORY_TIndexes: 3Since: 6.1

pi * ti * 0..m - Task historyinformation; must beenabled in the adminconsole

TASK_INST_CPROP_TIndexes: 2

pi * ti * 0..m DATA (1): binarycustom task property;optional

Task custom properties

TASK_INST_LDESC_TIndexes: 3

pi * ti * 1..m - Task descriptions indifferent locales

TASK_INSTANCE_TIndexes: 14

pi * ti - Human, administrative,and standalone tasks

TASK_MESSAGE_INSTANCE_TIndexes: 4

pi * ti * 0..f DATA (1..100): the taskinput /output/faultmessage of a humantask

Available for humantasks only (not foradministrative task)

TIDOCUMENTATION_TIndexes: 2

pi * ti * 0..f DOCUMENTATION(1..100): free-form text,optional

Available for humantasks only (not foradministrative task)

Escalation data (optional)

EIDOCUMENTATION_T pi * ei * 0..m DOCUMENTATION Escalation



Trademarks




Indexes: 2 (1..100): free-form text,optional

documentation

ESC_INST_CPROP_TIndexes: 2

pi * ei * 0..m DATA (1): binarycustom task property;optional

Escalation customproperties

ESC_INST_LDESC_TIndexes: 2

pi * ei * 0..m - Escalation descriptions

ESCALATION_INSTANCE_TIndexes: 8

pi * ei - Escalation instances

Variable data

SCOPED_VARIABLE_INSTANCE_B_TIndexes: 3

pi * 1..M DATA (1..100): BPELvariable data

Contains variable datamodelled in BPEL

VARIABLE_SNAPSHOT_B_TIndexes: 2

pi DATA (1..100): a copyof the process instanceinput and outputmessage

Preserves the processinstance input message

Authorization data

RETRIEVED_USER_TIndexes: 5

u * 1..M - Users with a permissionon objects

WORK_ITEM_TIndexes:

• 20-23since 6.1

• 10 in 6.0

(pi * ti) * 5 - Work items

WI_ASSOC_OID_TIndexes: 4Till: 6.1

(pi * ti) * 3 - Work items

Audit trail (optional)

AUDIT_LOG_TIndexes: 1

pi * ai * 0..m VARIABLE_DATA(1..100)EXCEPTION_TEXT (5)CORR_SET_INFO (1)ADDITIONAL_INFO(1..10)

Auditing data (notdeleted automatically),see Structure of theaudit trail databaseview for businessprocesses for moreinformation

TASK_AUDIT_LOG_TIndexes: 1

pi * ti * 0..m USERS ( 1..10)DESCRIPTION (1..10)MESSAGE_DATA(1..100)

Auditing data (notdeleted automatically),see Structure of theaudit trail databaseview for human tasksfor more information



Trademarks


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.websphere.bpc.620.doc/doc/bpc/rg5attbl.html




http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.websphere.bpc.620.doc/doc/bpc/rg5attbl_task.html





Periodic operations

During the lifetime of the BPEDB, access patterns, load, and size of the data in theBPEDB can change. Also, data files may get fragmented. To allow optimal dataaccess, run the following tasks periodically:

• Cleanup

• Maintain database files

• Maintain database statistics (do not run on empty databases)

• Database health check

Database health needs daily attention. As a rule of thumb, run the tasks described inMaintain database statistics and Maintain database files tasks regularly. Performcleanup of the database continuously or on a weekly basis; this depends on thesetup of the WebSphere Process Server environment. Note that cleanup is not runwith database tools - discuss this topic with the WebSphere Process Serveradministrator of your environment.

Maintenance tasks on database files and database statistics and cleanup typicallyrequire write access to the BPEDB. Often those are disk I/O intensive operations.You need to run them one after the other. Carefully plan these tasks and coordinatethem with the administrator.

Cleanup

For continuous good database performance, it is important that the size of theBPEDB is kept at a minimum level. The WebSphere Process Server environment isset up such that completed process instances and standalone human tasks areremoved. You can do this as follows:

• Automatically delete completed process instances and completedstandalone human tasks.

• Periodically delete completed business process instances and completedstandalone human tasks.

Script BPEDB_dataSubjectForDeletion_db-system.sql prints out thenumber of human tasks, process instances, staff query instances, and audit logentries that you can delete from the database using tools running on WebSphereProcess Server (Table 3).

Table 3. Deletion toolsTools for deletion Data



Trademarks




Cleanup service(configurable in the admin console sinceWebSphere Process Server V6.2, samplecleanup application provided for V6.0, V6.1)

Process instances and human tasks

deleteCompletedProcessInstances.py(Python script)

Process instances in end states

deleteCompletedTaskInstances.py(Python script)(since WebSphere Process Server V7.0)

Standalone task instances in end states

cleanupUnusedStaffQueryInstances.py(Python script)

Unused people resolution data

deleteAuditLog.py(Python script)

Audit log entries

BPC Explorer(admin client)

Manual deletion of process instances andstandalone human tasks

For a detailed description of the tools, see Cleanup procedures for BusinessProcess Choreographer.

Hints:

• Automatic deletion of process instances and standalone human tasks canbe configured at process and human task development time.

• After a script driven database cleanup, maintain database files andstatistics.

• Maintain statistics when a typical amount of records is in the BPEDB. Thismight be only the case before cleanup in some scenarios. In those cases,maintain statistics before the cleanup.

• After clean up, monitor the statistics for I/O and processor consumption. Ifthese show an increasing trend, this is an indication that you should runtasks Maintain database files and Maintain database statistics.

• For DB2 for z/OS, monitor for increases in getpages.

Maintaining database files

Database files store the data of database tables, as well as indexes on tables. Forgood database performance, it is important that the database files are physically wellorganized. The sample scripts (BPEDB_ maintainFiles_db-system.sql)contain sample statements for this operation.

Hint: After maintaining the database files, maintain the database statistics.

Maintaining database statistics



Trademarks


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6rxmx/index.jsp?topic=/com.ibm.wsps.602.bpc.doc/doc/bpc/tadmin_deleting_completed_process_instances.html

http://publib.boulder.ibm.com/infocenter/dmndhelp/v6rxmx/index.jsp?topic=/com.ibm.wsps.602.bpc.doc/doc/bpc/tadmin_staff_cleanup.html

http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/index.jsp?topic=/com.ibm.websphere.bpc.620.doc/doc/bpc/tadmin_log.html

http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.websphere.bpc.620.doc/doc/bpc/c2_cleanup.html




The database systems calculate the cost of data access (insert, update, delete, andselect statements) using available statistics, including:

• Table cardinality

• Cardinality and distribution of column values

• Index properties

• Others

For each database system described in this article, a sample script for maintainingthe database statistics (BPEDB_ maintainStatistics_db-system.sql) isprovided.

Database health check

Use the health and monitoring tools of your database system for this task. Regularchecks for alerts are mandatory to keep databases up and running.

DB2 for z/OS

The DB2 administrator can use DB2 system statistics to acquire insight on the healthof the DB2 subsytem and SQL processing. Look for increasing trends to determine ifnormal housekeeping activities or SQL tuning is required. Reorganize table spacesand index spaces after significant data modifications and growth.

You can gather DB2 statistics using the z/OS and DB2 trace facilities or extractingruntime statistics from the DB2 statement cache. A number of monitoring tools arealso available for this purpose. For more information, see DB2 9.1 for z/OS Usingtools to monitor performance.

DB2 LUW

You can use the DB2 Health Center to analyze and monitor the health of DB2databases.

With the Health Center, database administrators can define health indicators andalert notifications for resources such as table spaces. The database administratorwill be alarmed if any indicator signals that a database resource does not fit thedefined indicator anymore. For severe alerts and lock escalations, check thedb2diag.log file.

See the DB2 documentation for more information.

Oracle

To monitor database health, Oracle provides the following approaches: Oracle



Trademarks


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.perf/db2z_usetools2monitorperformance.htm




Enterprise Manager, and a set of database views and log files. The easiest way tomonitor the database health is to check the database home page in the OracleEnterprise Manager. It provides a comprehensive overview over the databasestatus. Starting from this, you can drill down into detailed views covering everyaspect of your database. Check the alerts and diagnostic summary regularly. ThePerformance tab is the starting point to identify performance problems.

For a reference to Oracle monitoring features, refer to the Oracle DatabaseAdministrator's Guide.

Performance

In a WebSphere Process Server environment, in which more and more processinstances are created, regular runs of the tasks described in the Periodic operationssection are important to keep the BPEDB at a good performing level. However,BPEDB performance can still degrade over time due to the following reasons:

• More users or applications are concurrently working on the system

• New or changed applications influence database access patterns

• New operations are implemented (such as backup runs during officehours)

• Infrastructure changes

• Others

To address performance issues coming from a changed workload or more data inthe BPEDB, memory analysis and SQL statement analysis are discussed in thissection. Also keep in mind:

• There may be other reasons for a degrading performance (parallelbackups, operating system problems, disk array controller problems,network problems, and so on).

• The user’s perception of a degrading database performance does notnecessarily indicate a technical problem with the database. For example,changed applications or slow networks can contribute to this perceptionas well.

Memory: Database systems provide statistics for memory usage. The mostimportant information related to this is the hit ratio. The hit ratio describes how oftenthe database reads the data from memory instead of reading from disk for a readrequest. Often, database environments have a hit ratio of more than 90%, so SQLperformance is not hit by I/O delays.



Trademarks




BPC internal SQL statements: These are used in process instance navigation,human task navigation, BPC API calls (except BPC API list queries), and accesssingle tables only. They perform well by default because they are not specific to acustomer environment. The indexes of the BPEDB schema allow optimal dataaccess. Response times are typically less than 10 milliseconds.

Complex SQL queries: Applications such as custom client applications runcomplex queries, issued through the BPC API, to present human task lists orprocess instance lists to the user. The resulting SQL queries often consist out ofseveral joins. Because those queries are specific to a customer environment,additional performance tuning may be needed by typically adding new indexes. Mosthuman task and process instance list queries need more than 50 milliseconds tocomplete.

Hints:

• Human task list and process instance list queries are not hit by databaselocking. They are run read only.

• Additional indexes introduce additional overhead for insert, update, anddelete operations.

See the database system specific sections for memory and SQL statement analysisand tuning, and general hints. More information about general WebSphere ProcessServer and BPEDB tuning operations is found at WebSphere Business ProcessManagement 6.2.0 Performance Tuning and in the performance related links in theResources section of this article.

DB2 for z/OS

DB2 performance is affected by a number of factors that needs to be monitored asthe data volume, applications, and number of users change.

The initial state of a new install should take into consideration with the assumptionsused by the DB2 optimizer in establishing SQL access paths. In most environments,this is a transient state whose duration will depend on the data growth and usage ofthe system. This is an important aspect of DB2 performance. Read through thefollowing sections carefully, and derive appropriate actions for your environment:

• Table statistics and SQL optimization: After the creation of the BPEDBtables, but before they are populated and RUNSTATS has been run, payattention to table data growth to insure proper SQL performance as datais accumulated. The Database Management System (DBMS) usesstatistics to determine the appropriate access paths to use for efficientSQL statements execution. For a new installation and before theaccumulation of data and the execution of RUNSTATS, default statistics



Trademarks


http://www.redbooks.ibm.com/abstracts/redp4551.html




are used for SQL optimization. The default statistics are adequate forSQL optimization until persistent data accumulates and invalidates thedefault values. Then, performance may degrade. When this occurs,RUNSTATS needs to be executed to gather factual statistics based onthe resident data in the tables. Proactive monitoring of data growth andrunning RUNSTATS when appropriate can avoid degradation inperformance due to invalid statistics.

• Empty tables: Complexity is introduced when tables are empty andRUNSTATS is run. In this case, RUNSTATS sets the statistics to zero.Zero statistics can result in sub-optimal SQL performance if large volumesof data actually exist. If, during execution, tables become populated withtransient data, performance problems may occur. These types of tablesare considered volatile: the amount of data they contain ranges from fewto many in a non-predictable way for the database administrator.Therefore, executing RUNSTATS will not provide useful statistics. In casethat this is a regular pattern in your environment, consider changing theVOLATILE option on specific tables and adjusting the NPGTHRSHparameter. See Volatile tables of Business Process Choreographer tableson DB2 for details.

• Object reorganization: Once the system has evolved beyond the initialstate, consider RUNSTATS as data volumes increase and data valueschange. RUNSTATS can help in determining if data needs to bereorganized to re-establish clustering or reclaim unused space that canresult from certain insert patterns based on ever increasing keys. To learnmore about determining when reorganization may be needed, see DB29.1 for z/OS Determining when an object requires reorganization.

• Output buffer size: Be sure to monitor the output buffer size to eliminateany log delays due to active log contention. Choose as large a size asyour system can tolerate to decrease the number of forced I/O operationsthat occur because there are no more buffers. A large size can alsoreduce the number of wait conditions. For more details, see DB2 9.1 forz/OS OUTPUT BUFFER field.

• Choose fast devices for log data sets: The devices assigned to theactive log data sets must be fast. Because of its high sequentialperformance, IBM Enterprise Storage Server® (ESS) is particularlyrecommended in environments in which the write activity is high. See DB29.1 for z/OS z/Architecture and the z/OS operating system for details.

• Avoid device contention: Place the copy of the bootstrap data set and, ifusing dual active logging, the copy of the active log data sets, on volumesthat are accessible on a path different than that of their primarycounterparts.

• Monitor buffer pool usage and hit ratio to size the buffer pool correctly.



Trademarks


http://www.ibm.com/support/docview.wss?uid=swg21425402


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.ugref/db2z_determineobjectreorg.htm


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.inst/db2z_ipf_outbuff.htm

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.inst/db2z_ipf_outbuff.htm

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.intro/db2z_zarchitectureandzos.htm




Avoid using buffer pool 0 since the DB2 catalog is assigned to that bufferpool. Also, at minimum, do not use the same buffer pools for indexes andtable spaces.

• APPEND option: In a data sharing environment, consider using the"APPEND=YES" option. See Using the APPEND option for BusinessProcess Choreographer tables on DB2 and DB2 9.1 for z/OS Memberaffinity clustering for details.

DB2 LUW

This section describes DB2 LUW specific operations for performance. Also, assess ifthe options as described in Volatile tables of Business Process Choreographertables on DB2 and Using the APPEND option for Business Process Choreographertables on DB2 need to be implemented in your environment.

Tools: You can use the Activity Monitor, as part of the DB2 Control Center, to get anoverview of the BPEDB database performance with a focus on SQL statements.Optionally, use the sample scripts provided below:

• BPEDB_enableMonitors_db2_luw.sql

• BPEDB_basicPerformanceAnalysis_db2_luw.sql

• BPEDB_disableMonitors_db2_luw.sql

Scripts: While a performance degradation of the BPEDB occurs, do the following:

1. Run the BPEDB_enableMonitors_db2_luw.sql script.

2. Wait a certain amount of time to get a good representation of theworkload (about 10 minutes). For the SQL statement lists, only thosestatements are shown that have been run more than 10 times.

3. Run the BPEDB_basicPerformanceAnalysis_db2_luw.sql script.

4. Run the BPEDB_disableMonitors_db2_luw.sql script.

The output of Step 3 (BPEDB_basicPerformanceAnalysis_db2_luw.txt) isexamined below:

Buffer pool hit ratio list (memory): This list shows the buffer pool hit ratios for allbuffer pools configured on the BPEDB (TOTAL_HR_PCT). If a low buffer pool hit ratiois detected, provide more memory to the related buffer pool, if possible. For a morecomprehensive analysis, refer to Buffer pool management.

Hint: If more memory is provided than the operating system supporting the DB2



Trademarks




http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.dshare/db2z_memberaffinitycluster.htm






http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=/com.ibm.db2.udb.admin.doc/doc/c0005391.htm



instance, the buffer pool will be disabled and performance degrades significantly. Inthis case, an entry into the db2diag.log file will be written.

Top SQL queries list (BPC internal statements and complex SQL queries): Thislist shows the SQL queries with the longest execution times. Typically, human taskand process instance list queries are found at the top of the list, especially if theapplication uses human workflow features and requires human interaction.

Top SQL updates, inserts, and deletes list (BPC internal statements): This listshows the SQL insert, update, and delete statements with the longest executiontimes.

The following sample scripts are described below:

• BPEDB_getTopStatements_db2_luw.sql

• BPEDB_recommendIndexes_db2_luw.cmd

• BPEDB_createRecommendedIndexes_db2_luw.cmd

• BPEDB_showAccessPlan_db2_luw.cmd

If SQL statements are found in the SQL statement lists that have run longer thanexpected (for example, a human task list query running for 6 seconds), steps can betaken to support a faster execution:

1. Run the BPEDB_getTopStatements_db2_luw.sql script.

2. Run the BPEDB_recommendIndexes_db2_luw.cmd script.

The output of the BPEDB_recommendIndexes_db2_luw.cmd script containsrecommendations for new indexes. Use theBPEDB_createRecommendedIndexes_db2_luw.cmd script to create theadditional indexes.

Hint:

• The DB2 Design Advisor also shows indexes that are not used for theSQL statements provided. These indexes must not be dropped. They areneeded for internal BPC statements.

• Carefully review the recommended indexes and assess its applicabilityfrom a database administrator point of view.

If more analysis on the problematic SQL statements is required, run theBPEDB_showAccessPlan_db2_luw.cmd script. The output of this script showsaccess plans of the top SQL statements and can be used as an entry point for a



Trademarks




more detailed analysis.

Oracle

By default, Oracle monitors the activity in the database continuously. The AutomaticDatabase Diagnostic Monitor (ADDM) reports performance problems of the last runon the database home page of the Oracle Enterprise Manager. Findings arereported in the Diagnostic Summary section. If findings are present, an ADDMPerformance Analysis section is also displayed.

• If you click the numeric value of findings, the ADDM page is displayed.This page shows a graphical outline of the database activities and theADDM findings.

• If you click on an ADDM finding, either on the database homepage, or onthe ADDM page, you can drill down into more details, such as the topSQL activities.

• If you review the top SQL activities, you can display the SQL statements,and you can follow the recommendations, such as running the SQLTuning Advisor.

• If you run the SQL Tuning Advisor, you can decide whether or not toimplement the recommendations. Before recommendations are applied,review them carefully.

Besides ADDM, Oracle Enterprise Manager allows you to review all relevantperformance figures on the Performance page. You can switch to the Performancepage by clicking the Performance tab on the database homepage. If you click acategory in one of the figures, for example User I/O, you can drill down into thedetails. The Additional Monitoring Links section provides links to details pages forseveral performance aspects. To identify the top SQL, click Top Activity. The TopActivity page shows the most time consuming SQL statements and the most timeconsuming sessions. If you click an SQL statement, you can review the details of thestatement including the access plan, and you can run the SQL Tuning Advisor.

Deadlocks

Reasons for deadlocks when running process instances and client applicationsusing the BPC API are grouped into two categories:

1. Application design (process model and client application)

2. Database configuration (missing indexes or statistics, lock escalations)



Trademarks




Examples for deadlocks resulting from application design:

1. Parallel branches of process instances use the same set of variables forwrite access in reverse order. In this case, deadlocks can occur on theSCOPED_VARIABLE_INSTANCE_B_T table. Hint: In Java™ snippets,variables are accessed for write by default. In case that only read accessis needed, declaring this avoids updates. See Using Java methods inprocess snippets.

2. A transaction started by the application spawns more than one BPC APIcall, and therefore, holds locks on many resources. Locks may getacquired in a way that leads to deadlocks. Changes to the application arethen required in most cases.

Examples for deadlocks resulting from the database configuration are:

1. Lock escalations: This is the conversion of several locks within the sameobject to a single higher level lock. For example, conversion from rowlocks to a table lock.

a. DB2: Increase the amount of resources available for locks(parameter LOCKLIST; see scriptBPEDB_initialSetup_db2_luw.sql). Lock escalations arelogged to the db2diag.log file by default.

2. The database uses a wrong index or performs a table scan when runningsimple SQL statements. Normally, this is related to outdated statistics.See Periodic operations on how to update the statistics.

Deadlocks lead to a rollback of one of the involved transaction. They can beclassified into:

• BPC internal transactions: In general, they will be retried immediately.In case a transaction does not succeed after several retries, a message isput into the hold queue. See Managing failed events for details.

• Transactions started by API calls: An exception will be thrown in theapplication client code. Client-side exception handling can include a retrystrategy.

Deadlocks can be accepted as long as they do not significantly impact the overallsystem performance and stability.

Lock timeouts are sometimes interpreted as deadlocks, due to similar exceptionmessages. Typically, lock timeouts occur when the database is overloaded, or if the



Trademarks


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/topic/com.ibm.wbit.620.help.bpel.ui.doc/concepts/cjvameth.html


http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/index.jsp?topic=/com.ibm.websphere.wps.620.doc/doc/recovery/cadm_failedoverview.html



database is not well configured. See the Initial setup section and perform the actionsdescribed in Periodic operations.

Hint: In rare cases, database statistics must be manually adjusted. See thedatabase system documentation for more details.

DB2 for z/OS

You can improve locking in a highly concurrent environment by setting theRRULOCK option in DSNZPARM. Set this option to YES to reduce the probability ofdeadlocks in some situations. This option becomes more important as the userconcurrence levels increase. For more information about this parameter, see DB29.1 for z/OS U LOCK FOR RR/RS field.

Conclusion

This article discussed the BPEDB database of WebSphere Process Server, whichstores business processes and human tasks. Details on its structure and accesspatterns have been provided. Maintenance procedures, performance tuning, andinformation about cleanup tasks round off the database administrator's view of theBPEDB.

This article is part of an article series on WebSphere Process Server operations:

• Part 1 describes essential tasks for system administration and operationof a healthy WebSphere Process Server production environment. Itcovers various approaches and tools.

• Part 2 describes the options that are available to cleanup completedinstances from the Business Process Choreographer database anddiscusses application design patterns for an optimal BPEDB size. Futurearticles in this article series will cover such topics as applicationmanagement, monitoring, and repairing or archiving options.



Trademarks


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.inst/db2z_ipf_rrulock.htm

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.inst/db2z_ipf_rrulock.htm

/developerworks/websphere/library/techarticles/0912_herrmann1/0912_herrmann1.html

/developerworks/websphere/library/techarticles/0912_herrmann2/0912_herrmann2.html



Downloads

Description Name Size Downloadmethod

Sample scripts BPEDB_scripts_db2_zos.zip XXKB HTTP

Sample scripts BPEDB_scripts_db2_luw.zip 17KB HTTP

Sample scripts BPEDB_scripts_oracle.zip 9KB HTTP

Information about download methods



Trademarks


http://public.dhe.ibm.com/software/dw/wes/0912_grundler/BPEDB_scripts_db2_zos.zip

http://public.dhe.ibm.com/software/dw/wes/0912_grundler/BPEDB_scripts_db2_luw.zip

http://public.dhe.ibm.com/software/dw/wes/0912_grundler/BPEDB_scripts_oracle.zip

http://www.ibm.com/developerworks/library/whichmethod.html



Resources

• WebSphere Process Server 6.2: Structure of the audit trail database view forbusiness processes

• WebSphere Process Server 6.2: Structure of the audit trail database view forhuman tasks

• DB2 9.5 Buffer pool management

• Options for maintaining an optimal Business Process Choreographer databasesize

• WebSphere Process Server 6.2: Cleanup procedures for Business ProcessChoreographer

• WebSphere Process Server 6.1: Using scripts to administer Business ProcessChoreographer


• Using scripts to administer Business Process Choreographer


• White paper: Business Process Choreographer - Process Cleanup ServiceSample V2

• WebSphere Process Server 6.0-7.0: How scaling WebSphere Process Server,and especially Business Process Choreographer, can impact business processclient architecture

• WebSphere Process Server 6.0-7.0: Business Process Choreographer Explorer- Customization and Tuning Options

• WebSphere Process Server 6.0-7.0: Performance Tuning Worksheet forBusiness Process Choreographer

• WebSphere Process Server 6.0-7.0: Improving the performance of complexBPC API queries on DB2

• WebSphere Process Server 6.2 and later: WebSphere Business ProcessManagement 6.2.0 Performance Tuning

• WebSphere Process Server 6.2 and later: Leading Practices for WebSphereDynamic Process Edition V6.2

• WebSphere Process Server 6.2 and later: PA71: WebSphere Process Server -Query Table Builder



Trademarks







http://www.ibm.com/developerworks/websphere/library/techarticles/0912_herrmann2/0912_herrmann2.html

http://www.ibm.com/developerworks/websphere/library/techarticles/0912_herrmann2/0912_herrmann2.html



http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r1mx/topic/com.ibm.websphere.bpc.610.doc/doc/bpc/tadmin_scripts.html

http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r1mx/topic/com.ibm.websphere.bpc.610.doc/doc/bpc/tadmin_scripts.html



http://publib.boulder.ibm.com/infocenter/dmndhelp/v6rxmx/index.jsp?topic=/com.ibm.wsps.602.bpc.doc/doc/bpc/tadmin_scripts.html






















• WebSphere Process Server 6.2 and later: Using the Query Table Builder inWebSphere Process Server V6.2

• WebSphere Process Server 6.1 and earlier: WebSphere Process Server 6:Business Process Choreographer - BPC Queries - Performance TuningMethodology For DB2

• WebSphere Process Server 6.1 and earlier: WebSphere Process Server V6.1 –Business Process Choreographer: Performance Tuning Automatic BusinessProcesses for Production Scenarios with DB2

• WebSphere Process Server 6.2: Using Java methods in process snippets

• WebSphere Process Server 6.2: Database views for Business ProcessChoreographer

• Managing failed events

• Business Process Management samples and tutorials

• Volatile tables of Business Process Choreographer tables on DB2

• Using the APPEND option for Business Process Choreographer tables on DB2

• DB2 9.1 for LUW Information Center

• DB2 9.1 for z/OS Information Center

• DB2 9.1 for z/OS data structures

• DB2 9.1 for z/OS data sharing

• IBM Redbook: DB2 9 for z/OS Technical Overview

• DB2 9.1 for z/OS Using tools to monitor performance

• DB2 9.1 for z/OS Determining when an object requires reorganization

• DB2 9.1 for z/OS OUTPUT BUFFER field

• DB2 9.1 for z/OS Inserting rows at the end of a partition

• DB2 9.1 for z/OS member affinity clustering

• DB2 9.1 for z/OS U LOCK FOR RR/RS field

• DB2 9.1 for z/OS active log data set parameters: DSNTIPL

• DB2 9.1 for z/OS z/Architecture and the z/OS operating system

• Oracle 11g documentation library

About the authors



Trademarks


http://www.ibm.com/developerworks/websphere/library/techarticles/0909_grundler/0909_grundler.html

http://www.ibm.com/developerworks/websphere/library/techarticles/0909_grundler/0909_grundler.html










http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r2mx/index.jsp?topic=/com.ibm.websphere.wps.620.doc/doc/recovery/cadm_failedoverview.html

http://publib.boulder.ibm.com/bpcsamp/




http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc/db2prodhome.htm

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.intro/db2z_db2datastructures.htm




http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/topic/com.ibm.db29.doc.inst/db2z_ipf_outbuff.htm

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.admin/db2z_insertrowsatend.htm


http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/topic/com.ibm.db29.doc.inst/db2z_ipf_rrulock.htm



http://www.oracle.com/pls/db111/homepage



Jonas GrundlerJonas Grundler is a Business Process Management Architect,focusing on performance and database. He has worked with BPC since2002. After being a consultant in IBM Software Services for WebSphere(ISSW) for more than 3 years, Jonas joined the WebSphere BPMDevelopment and Architecture Team in 2006.

Rolf BaeurleRolf Baeurle works in the WebSphere Business Process ManagementDevelopment Team at the IBM Research and Development Lab inBoblingen, Germany since 2002. His responsibilities are database layerdesign and implementation.

David J. BonaccorsiDavid P. Bonaccorsi is the Architecture Lead for Business ProcessManagement on System Z products and solutions. He has been in thisrole since 2005. David's responsibilities include BPM V7 for System ZIndustry Solutions and Performance testing. He and his team haveproduced five generations of WebSphere Prcess Server, WebSphereEnterprise Service Bus, and BPM performance reports. He has over 33years of development experience as an IBM Business Partner and IBMSoftware Group development member with a broad background in IBMmiddleware, including the WebSphere MQ Family of products and IBMindustry solutions.

Andreas Fried



Trademarks


http://www.ibm.com/developerworks/websphere/services/



Andreas Fried is a Software Engineer at the IBM Research andDevelopment Lab in Boblingen, Germany since 2002. He initiallyworked on WebSphere Portal with focus on Web Services for remoteportlets (WSRP). He joined the WebSphere Business ProcessManagement Development Team in 2005, introducing the BusinessProcess Management samples page. He is currently working on thedatabase access layer of Business Process Choreographer with afocus on Oracle.

Fred OroscoFred Orosco is a Senior Software Engineer working in the DB2development organization. He joined IBM in 1976 and has worked in awide variety of areas within the DB2 organization over his career. Hisfocus over the past few years has been in the integration testing of DB2with the WebSphere family of products with a focus on DB2performance tuning.

Michael PoirierMichael Poirier is a Senior Software Engineer, focusing on BusinessProcess Management performance and architecture for the z/OSplatform. He has been working with WebSphere Process Server since2005. Mike has worked in IBM Software Group since 1996, working onWebSphere, CICS, and WebSphere MQSeries.



Trademarks




operating a websphere process server...

Documents