planning optimal lotus quickr services for portal (j2ee) deployments

Component to adjustValue

For each Portal Server in your cluster do the following:Set Transport buffer size to 180 MB

For the Deployment ManagerSet Transport buffer size to 180 MB

For each node agent do the followingSet Transport buffer size to 180 MB

Machine FunctionMachine Specifications(All Systems are IBM Blades connected via fibre to a SAN, FATtT)

Deployment manager2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

Quickr Server 2 separate nodes

2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

DB2 server2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

LDAP server2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

WebServer2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

ParameterWindows 2003Linux

KeepAliveTimeout10 seconds10 secondsA high timeout value can increase contention for http server processes, if you are running out of http processes decrease this value

ThreadsPerChild102425The higher value on windows systems is due to a different process model for windows

MaxKeepAliveRequests100100Selecting 0 here will allow an unlimited number of requests on a single TCP connection

MaxRequestsPerChild00

StartServersN/A2


ThreadLimit102425

ServerLimitN/A80

MinSpareThreadsN/A25

MaxSpareThreadsN/A2000

MaxClientsN/A2000Do not allow users to queue up waiting to get to the http server.

Access LoggingonoffThis can be turned off by commenting the line:CustomLog logs/access_log common in httpd.conf

Planning Optimal Quickr Deployments

Presented by:Norman McBrideSVT Reliability,Dublin Software Lab

Agenda For This Session

Planning and Preparing for Your Deployment.

Installation Options and Deployment Topologies

Performance and Tuning Considerations.

The Basis for Our Recommendations?

Basic Troubleshooting Techniques.

Planning and Preparing for your Deployment.

Planning is an essential step:

If you Fail to Plan then Plan to Fail!!

Planning should be based on the intended short term use of the deployment and the possible future long term use of the system to avoid future performance and reliability issues and possible re-deployment at a later stage.

Some items to consider during Planning

The purpose of the system?

Hardware and Software considerations.

What is the intended user population size? How many concurrent users will be 'on' the system?

Consider what the intended user population will primarily use the system for, will the primary use be document centric or Wiki Centric or a mix.

What are the criteria for availability?

What is the criteria for user response times?

The Number of TeamPlaces that will potentially be on the system.

Place Design and Template design.

Version 8.1.1 has new functionality such as ECM and new connectors integration. Plan to upgrade from previous versions if doing a new deployment. 8.1.1 is the current recommended level.

Machine Allocation for Your Deployment.

In an ideal situation each component/node in a deployment should reside on its own machine.

During SVT Reliability testing we have each component installed on a single high spec machine.

If co-location of some components is unavoidable it is preferable that at a minimum the Database server should be installed on separate hardware.

Co-location of components, whilst feasible is not advisable in a production environment.

Separate systems helps to avoid resource contention.

Co-located servers will compete for resources.

Minimum Hardware requirements

We would recommend at a minimum hardware with the following specifications:

Absolute Minimum 4 GB free disk space for installation for Lotus Quickr.

Under minimal load, Lotus Quickr can function with 2GB of RAM. However, 4GB is an optimal starting point for RAM in a production environment.

CD-ROM/DVD-ROM drive

Processor: CPU speeds of late-model, mid-range to high-end servers are recommended. Pentium 1 GHz or equivalent at a minimum. Production environments should consider the Pentium 4 processor at 2.5 GHz or higher. SVT test Systems used Dual-Core Intel Xeon processors.

Quickr can take advantage of multi-CPU systems therefore if possible it should be deployed on systems with these processors.

Optimal Hardware as used by SVT in our Test Environment

Software Requirements

The following Operating Systems are supported:

Microsoft Windows 2003 Enterprise Edition and Standard Edition with SP2

Microsoft Windows Server 2003 R2 on 32-bit only

Red Hat Enterprise Linux (RHEL) Enterprise Server (ES), Advanced Server (AS).

We would advise where possible that a 'dedicated' OS is provided for each Lotus Quickr component.

Detailed Hardware and Software requirements can be found at the url below, it is advisable to check these in detail during the planning stage:

http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27014510

Planning Considerations

Considerations for building your environment

Web server considerations

Database considerations

Security considerations

Clustering considerations

Other considerations

Considerations when building your Environment
You should consider the following when building a staging/integration or a production environment

Planning for integration and staging environments

Environment should mimic the intended production environment.

Planning for a production environment:

Environment should ideally be designed to provide for redundancy and failover.

Should be designed with criteria considerations in mind.

Web server considerations

Do you have an existing Web server that you want to use?

Yes:Ensure existing HTTP server is at supported level.

No:A Web server is not required by Lotus Quickr, However an external web server is recommended.

Database Planning

When installing Lotus Quickr, you can choose one of the following general approaches for your database software:

Install and use Lotus Quickr with the default OOTB DB2 database:

This is not recommended for a high load production environment.

Install with the default DB and then transfer the database information to a remote DB:

Allows you to begin working with Lotus Quickr without additional DB deployment or if you wish to wait before integrating the remote DB into your environment.

By default, IBM Lotus Quickr 8.1 installs and uses IBM DB2 Universal Database Enterprise Server Edition version 9.1fp4a. Installing with DB2 lets you quickly get Lotus Quickr installed and running in a production-level environment.

When you plan to transfer data to a remote database, perform the database transfer before you use Quickr extensively.

Large amounts of data in the databases can cause the database transfer to fail if your Java heap size is not large enough.

Apart from the potential for transfer failure, large databases can take a considerable amount of time to transfer.

Perform the database transfer as soon as it is practical to avoid bottleneck problems in a production environment.

SVT DB Experiences

During SVT Reliability testing we have always installed our database software on a remote system and performed our reliability testing on deployments with remote databases. We have found this to be the optimal deployment method.

LDAP/User Registry considerations

It is recommended to always have security enabled otherwise certain applications may not function properly.

After a user has been authenticated, the system can determine if that user is authorised to access the resources that are requested. Security is resource driven and users need to be allowed permission to access specific resources on the system.

A user registry holds user account information, such as a user ID and password that can be accessed during authentication. IBM WebSphere Application Server and IBM Lotus Quickr support multiple types of user registries, the most commonly used being:

Lightweight Directory Access Protocol (LDAP) user registry.

Other Common Security Considerations

Do you intend to use SSO(single sign-on)/why would I want to use it?

Single sign-on is used to provide a secure method of authenticating a user one time within an environment and using that single authentication (for the duration of the session) as a basis for access to other applications and systems. For example if Lotus Quickr is to be integrated with an ECM backend system SSO can be used to provide a simpler user experience removing the necessity for multiple log on on each system.

Do you intend to use SSL?(secure socket layer)/Why would I want to use it?

Configuring Lotus Quickr for SSL adds security to the client-portal exchange. It encrypts all traffic between the client browser and the server, so that no one can "eavesdrop" on the information that is exchanged over the network between the client browser and Lotus Quickr.

General Clustering Considerations

Clustering is the concept of distributing the workload across multiple machines.

Why use clustering? - For redundancy, scalability, performance, failover, and simpler maintenance.

A Lotus Quickr cluster is essentially a collection of multiple Lotus Quickr servers that are identically configured.

The deployment manager node should be installed separately before the cluster is configured.

You must install and configure Lotus Quickr separately on each node. In addition, there can only be one Lotus Quickr cluster per cell.

Do not use spaces in the cluster name or the cluster member name.

For the deployment manager and each Lotus Quickr node to be in the cluster, verify that each system clock is set to within 5 minutes of the others or the addNode command will fail.

In a clustered environment the 'Search Indexes' need to be located in a shared location that is used by each of the nodes.

Operating System Preparation

Windows

Verify the Operating System & service pack version and upgrade as necessary.

Check for a configured, fully-qualified host name.

Each System in the deployment should have valid hostname that can resolve to a valid static IP address.

Before installing Lotus Quickr, check that the system logon user ID you will use during installation has the necessary permissions and rights.

Verify that there is adequate space on the destination drive.

Quickr creates a DB user during install, this users password must be consistent with the security policies that are in place on the machine where installation is taking place. .

Disable any firewall products.

Ensure that the time and date settings on each system that will be used in the deployment are correct and consistent across all machines in the deployment. Also ensure that the regional settings are the same.

To avoid problems related to excessively long paths names, consider the following recommendations when installing -

Use a short installation path. For example, use C:\Quickr instead of C:\Program Files\IBM\Quickr.

Keep node names as short as possible.

Operating System Preparation

Linux

Verify the Operating System & service pack version and upgrade as necessary.

Check for a configured, fully-qualified host name.

Each System in the deployment should have valid hostname that can resolve to a valid static IP address.

Before installing Lotus Quickr, check that the user ID you will use during installation has the necessary permissions and rights.

Verify that there is adequate space on the destination drive.

Quickr creates a DB user during install, this users password must be consistent with the security policies that are in place on the machine where installation is taking place.

Disable any firewall products.

Ensure that the time and date settings on each system that will be used in the deployment are correct and consistent across all machines in the deployment. Also ensure that the regional settings are the same.

Operating System Preparation Linux cont'd.

Planning Linux disk space:

You should carefully plan in advance for the size of your file system in order to avoid related problems. The following disk space is the minimum required for each directory:

/Must have 1.5 GB or more available.

/optMust have 4 GB or more available.

/homeMust have 700 MB or more available.

/tmpMust have 600 MB or more available.

The default install directory for Lotus Quickr on Linux is '/opt'

The above are minimum recommendations, depending on your particular scenario you may require more space. Carefully consider the space allocations on your DB2 server as the database size will grow as content is added.

Setting the File Descriptor Limit:

Prior to installing Lotus Quickr, set the file descriptor limit to 40000. You can set this by issuing the command 'ulimit -n 40000'. You can check this setting by issuing the command 'ulimit -a'.

Installation program considerations:

To use the graphical user interface that is provided by the installation program you will need to install and configure X server on Linux (for example X-Windows or GNOME or KDE).

Operating System Preparation Linux cont'd.

Enabling Document Conversion Services:

To use document conversion services make sure that you have the appropriate package installed for your system. Enabling document conversion requires some extra steps on the Linux Platform.

For more detailed information on enabling Document conversion services see the infocenter at the following url:

http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/index.jsp?topic=/com.ibm.lotus.quickr.admin.wpv81.doc/wpf/dcs_info.html

Section Summary

Planning and Preparing for your Deployment. - What have we covered in this Section?

The importance of Planning BEFORE beginning to deploy.

Items to consider during Planning.

Machine Allocation considerations.

Minimum Hardware requirements.

Optimal Hardware as used by SVT in our Test Environment.

Software Requirements.

Specific Planning considerations:

The type of environment: Staging or Production

Web Server considerations.

Database considerations

Security Considerations

Clustering Considerations.

Operating System Preparation:

Windows & Linux.

Installation Options and Deployment Topologies

Installation Options

The Express Install

The installation process places all components on one node. With previous products, had to install individual components and then the 'main' product.

This method of install has the fastest install path.

This install type is suitable for evaluation or demonstration purposes or light departmental usage.

Websphere Application Server and WebSphere Portal server and DB2 database server are installed on the same system.

Advanced Single Server

This is referred to as the 'Typical' install option.

Use this option if you plan to expand the deployment into a cluster at some point.

Use this option if you want to configure the DB user ID.

Use this option to install a Single Server configuration with remote DB and remote Web Server.

This install is similar to the express install in that all the components are installed on the same machine.

Single Server With Remote HTTP/DB

Install using the 'typical' option.

Provides greater capacity and performance as the DB and web server are remotely located.

Quickr uses jdbc type 4 drivers, These are direct to database drivers that allow the local db2 to be disabled after db transfer to a remote system.

Can be used where there is a plan to expand into a cluster at a later point.

May be suitable for applications where failover and redundancy are not a requirement.

May be suitable for small to mid range deployments where clustering is not required yet.

Typical Two Node Cluster Configuration with remote HTTP and Remote DB

A Clustered configuration is recommended for production environments.

Install using the Advanced Enterprise Cluster option

A Cluster provides Load Balancing between the nodes.

Fail-over is provided for. If one server fails during a user session then the user session is not lost but 'fails-over' to the other node.

Redundancy is also provided in a clustered topology. This allows one node to be stopped for maintenance while the other node continues to provide service.

Performance is increased and the system can handle large loads better.

Cluster Deployment - Typical SVT Process

The typical (SVT reliability) process for installing a two node cluster with a remote DB2 server and remote web server is as follows:

Install and prepare the deployment manager.

Install the primary node.

Install the secondary node.

Install the DB server on a remote machine.

Transfer the Database to the remote Database server.

Enable Security against an external Directory.

Install and configure the remote http server and generate plugin file for http server.

Configure Search for the cluster.

System Tuning.

System Backup

Additional Deployment Options

IBM Lotus Quickr supports integration with other offerings within the Lotus Portfolio - such as Sametime and Connections.

IBM Lotus Quickr also supports a number of Third Party options such as:

Netegrity SiteMinder 5.5 and 6.0

Tivoli Access Manager (TAM) 5.1 and 6.0

Forward and reverse proxy servers via:

Apache HTTP Server 2.0.54

WebSphere Edge Server 6.0

Section Summary

Install Options and Deployment Topologies - What have we covered in this Section?

Deployment topologies:

Express install.

Advanced Single Server(Typical)

Advanced Single Server with remote HTTP and remote DB.

Typical SVT cluster deployment Process.

Advanced Enterprise Cluster with remote HTTP and remote DB.

Additional Deployment Options.

Performance and Tuning Considerations

General Performance Recommendations

For optimal performance a Cluster Configuration is recommended:

A cluster will provides Load Balancing.

There is redundancy in a Clustered configuration.

Fail over is catered for allowing for session preservation in the event of a node failing.

In Terms of Hardware configuration:

The ideal configuration for Lotus Quickr is at least Dual Processor systems.

The Database server should have a highly efficient Disk System.

This should be a dedicated internal disk array or a properly configured external disk array ideally connected via fiber.

Platform Selection:

Memory management is generally superior on Linux.

This allows for a larger Java heap size thus supporting more concurrent users.

I/O management is more efficient on Linux.

Performance Considerations - Installation

It is recommended to upgrade to the 8.1.1 build as significant performance improvements have been made over previous releases:

Reduction in the load on the database and directory servers.

Improved response times for specific operations.

Better Cluster scalability.

ECM integration.

Better connections integration.

Ensure that all applicable Quickr fixes available on fix central are applied. (See the references section at the end of the presentation for location information)

It is recommended to 'break out' the Lotus Quickr DB domains into separate databases. Four separate databases is considered to be optimum for basic Quickr deployments running under production level user loads:

Performance Recommendation for a Clustered Configuration

Lotus Quickr Servers2 Processor/4 GB Memory

Database Server4 processor8 GB MemoryDedicated Storage or disk array

Directory Server2 processor4 GB MemoryDisk array

HTTP Servers2 Processor4 GB Memory

Load Balancer2 Processor4 GB Memory

Cell

Deployment Mgr2 Processor4 GB Memory

System Specifications in SVT Reliability

Performance Considerations

It is essential to perform regular maintenance on databases to ensure that they perform well.

Use the automatic maintenance function in DB2 to perform regular maintenance.

The key operations to automate in DB2 are:

Defragmentation (REORG)

Optimizing Access (RUNSTATS)

Performance Considerations (cont'd.)

In a cluster the Search Indexes are stored in a share and mounted as a mapped drive on each node.

The search indexes should ideally be placed on High speed disk such as FAStT storage server.

Text Indexing should be run frequently

The default indexing time is every 30 minutes:

'jcr.textsearch.indexmaintenance.interval = 30'

This interval can be modified by altering the property shown above in the file icm.properties in the directory \Quickr\PortalServer\jcr\lib\com\ibm\icm

The default time should be acceptable for most installations.

In a cluster environment only one node needs to run indexing.

The indexer cannot be scheduled to run at specific times, only at intervals.

Usability Performance Tips

You should encourage users to exploit the Favourites feature for quick access to Teamspaces.

Encourage users to bookmark their most frequently used pages.

Manage your Teamspaces:

A lot of Public Teamspaces can make My Places awkward to use.

Does a Teamspace have to be made Public?

Ensure there is a policy for inactive places.

Ensure there is a monitoring policy in place for Place sizes.

Do not create a Place for everything, consider who should be allowed to create places.

Manage content distribution efficiently, consider the folder structure that you will use in libraries. For example on your OS file system would you place 15,000 documents into a single folder?

Consider the impact of theme customisation: seemingly small theme changes can have a significant performance impact.

Create templates for your users based on requirements.

Consider page design when creating custom places or templates.

You should ideally place one component on each page:

So for example have one page for a library, another for a wiki.

Do not place a group of components on the same page, for example a library, a wiki, a calendar and a blog all on the same page.

This is an important consideration for the most commonly 'hit' pages.

It is preferable to have more pages in a place than to have multiple components on each page.

There is a nightly sensor task that runs to check the size of each Teamspace.

Upgrade to the latest applicable WAS FP level, check here for more information on this:

http://www-01.ibm.com/support/docview.wss?rs=3264&context=SSVGKL&context=SSKU2L&uid=swg27013353&loc=en_US&cs=UTF-8&lang=en

Usability Performance Tips

Tuning

Much of the tuning that was developed and refined during testing is now implemented by default.

Our tuning is based upon measurements collected during testing in our controlled lab environment.

Customers should firstly apply the tuning steps as outlined in the Tuning Guide which is available from the following url: http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27010632

Our SVT tuning is particular to our environment and may or may not be suitable for yours.

It is important to remember that tuning and capacity are affected by a wide variety of factors including the workload on the system and the overall environment.

Our tuning steps may not be applicable in your environment and we are not recommending them. The purpose is to make users aware of the parameters used in our configuration.

When tuning your systems it is important to establish a baseline and monitor performance metrics to determine if any parameters need to be changed. When a change is made performance should be monitored to establish the effectiveness of the change.

SVT apply the following tuning to our systems before each test run:

Portal and Application Server Tuning.

DB Tuning.

HTTP server Tuning.

Operating System and Network Tuning

Tuning (cont'd.)

Adjusting the JVM Heap Size

The value of the JVM heap size is directly related to the amount of Physical memory on the system. The JVM heap size should never be set higher than the amount of physical memory on the system.

When setting the JVM heap size for a server the following points should be kept in mind:

Ensure that the system has adequate main memory for all the processes running on the system and for the Operating system.

After making any changes to the Heap size it is advisable to monitor the system to see that paging is not occurring.

32 bit Operating Systems have an address space limit of 4Gbytes.

In addition to this limit some Operating Systems restrict the size of processes to be even less than the limit. Greater detail on how this applies to Windows systems can be found here: http://support.microsoft.com/default.aspx?scid=kb;en-us;555223

On our SVT deployments we have enabled the '3G' switch to circumvent this problem. This switch can be enabled by the addition of the /3G flag in the boot.ini on a Windows system.

See here for more info on the 3G switch: http://technet.microsoft.com/en-us/library/bb124810.aspx

If the JVM process grows to a size larger than that allowed by the Operating System then the process can terminate suddenly with unexpected consequences.

Tuning (cont'd.)

Adjusting the JVM Heap Size - How to set?

On a Single server, login @ https://host.domain:10039/admin

On a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp

Select Servers, Application Servers on the left hand side and then select WebSphere_Portal.

Server Infrastructure select Java and Process Management and then select Process Definition

Now under the section heading "Additional Properties" click on " Java Virtual Machine".

Modify the value "Maximum Heap Size" as required based on the table below:

Tuning (cont'd.)

Thread Pool Tuning

The values that we use here are those that we have used in our reliability test runs. We have achieved consistent results with these values in our environment within the constraints of our particular scenario.

Monitor the results of any adjustments to these values and adjust the values based on observations such as increasing WebContainer values if the servlet threads are busy most of the time.

Adjusting Thread Pool sizes - How to set?



Under the section titled "Additional Properties" select the section "Thread Pools".

Modify each of the threadpool settings so that the maximum value = Minimum value as shown in the screenshot below. If there is no "Thread inactivity timeout" a value will be required, use 3500 for this value.

Tuning (cont'd.)

Transport Buffer Size Tuning

Adjusting the Transport buffer sizes can lead to improved data replication performance in a clustered environment.

Adjusting Transport Buffer sizes - How to set?


1. For each Portal server:


Under Additional Properties select Core Group Service and then Transport Buffer Size.

2. For each Node agent:

Navigate to System administration > Node agents > node_agent_name > Core group service

3. For the Deployment Manager:

Navigate to System administration > Deployment Manager > Core group service

Tuning (cont'd.)

DataSource Connection Pool Tuning

The Lotus Quickr Database is used to hold information for the various applications in Lotus Quickr. In our SVT Reliability environment we have found the settings presented here to provide optimum results based on our scenario in our environment.

Adjusting Connection Pool Settings - How to set?



Select Resources, JDBC providers.

Now select wpdbJDBC_db2 from the resulting screen.

Now select Data sources from the resulting screen.

Now select wpdbDS from the resulting screen.

Now select Connection pool properties from the resulting screen.

On this screen you need to change the Maximum connections to 120.

Select OK and then save the changes.

Tuning (cont'd.)

Database Tuning

Lotus Quickr uses databases for its core functionality. In our controlled environment we used a single database for all content.

As mentioned earlier we would recommend using a remote database server and also to 'break out' the Lotus Quickr DB domains into separate databases in a production environment.

We used the configuration specified in the Infocenter for database setup @ http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/topic/com.ibm.lotus.quickr.admin.wpv81.doc/wpf/setup_db2.html

While the Quickr database is configured for high capacity performance it is possible that various tuning adjustments will be necessary from time to time. These tuning adjustments are based on the amount of database traffic and the size of the table populations

In addition to the to the recommendations in the InfoCenter we adjusted the following settings. These settings can all be applied from a DB2 CLP:

db2set DB2_ASYNC_IO_MAXFILOP=512 (to set the DB2 registry variable).

db2set DB2_USE_ALTERNATE_PAGE_CLEANING=on

db2set DB2_INLIST_TO_NLJN=YES

db2set DB2_EVALUNCOMMITTED=YES

db2 "update db cfg for wpsdb using MAXFILOP 512" (This adjustment allows for additional database files to be allowed opened)

db2 "update dbm cfg using JAVA_HEAP_SZ 4096" (Adjust the heap size for the DB2 JVM)

Tuning (cont'd.)

Database Tuning

We also made the following Bufferpool adjustments, again please note these are entirely subjective and dependent on the volume of data in the database and the workload on the server, these are the settings that we used in our environment.

As the volume of content in the database increases it may be generally be necessary to increase the size of the bufferpools.

We also created the following index - Note this is now included by default in 8.1.1, via JCR PK60132

DB2 CREATE INDEX JCR.ICMSTJCRLINKS_IXQ81 ON JCR.ICMSTJCRLINKS (TVID ASC, TIID ASC) ALLOW REVERSE SCANS

Tuning (cont'd.)

Recommended Database Maintenance

There are two database attributes that DB2 relies upon to perform optimally. These attributes are the database catalog statistics and the physical organisation of the data in the tables.

Catalog statistics should be recomputed periodically.

It is recommended that such maintenance is performed during periods of low demand or off peak hours or if possible when the system is offline.

The DB2 'runstats' command is used to count and record the statistical details about tables, indexes and columns.

The format we use on our database is of the form:

db2 -x -r "runstats.db2" "select rtrim(concat('runstats on table ', concat(rtrim(tabSchema), concat('.', concat(rtrim(tabname),' on all columns with distribution and detailed indexes all allow write access'))))) from syscat.tables where type='T'"

The command above generates a script 'runstats.db2' which contains the computed statistics on all the database tables, then execute that on that script as per the command below.

db2 -v -f "runstats.db2"

Tuning (cont'd.)

Recommended Database Maintenance

To determine which tables in the DB might benefit from re-organisation (re-org) we can use the command:

db2 reorgchk current statistics on table all > "reorgchk.txt"

For those tables that require re-organisation you can use the following command to re-organise the table based on its primary key.

db2 reorg table tableschema.tablename

To perform a re-org on all tables you can use the command:

db2 reorgchk update statistics on table all

Database performance is critically important for obtaining good performance from Lotus Quickr.

We have found that the maintenance practices that we have outlined here have been crucial to the performance of Lotus Quickr in our environment. Additional database maintenance may be required in your production environment.

It is possible to schedule these maintenance tasks such as table re-organisation.

Additional information on DB2 administration, tuning and monitoring can be found in the infocenter @ http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp

Tuning (cont'd.)

Web Server Tuning

Our SVT Reliability configurations were tested using IBM HTTP Server 6.0.2.29. It is advisable to have the HTTP server remotely located to avoid resource contention with other servers.

In our test environment we had a dedicated LAN with clients and servers located on the same segment. This meant that we had high bandwidth and low latency.

In real work networks bandwidth is limited and latency can be significant.

There are some steps that can help reduce transaction times:

Compress content on the http server using gzip compression.

Allow client side caching of images, javascript files and stylesheets.

Much of the content that is server by Quickr can be compressed to reduce transmission time and save network bandwidth.

IBM HTTP server supports compression through the mod_deflate module. When enabled, the http server will check the 'Accept-Encoding' header sent by the browser to see if the browser can accept a compressed version of the content. If it can then the http server will compress the content before sending it to the browser.

Whilst the reduction in network traffic from enabling compression can be significant there is a trade off as compression on the http server does result in increased processor utilisation on the http server.

The process for enabling gzip compression can be viewed at this link: http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg21289555

Tuning (cont'd.)

Web Server Tuning

Another method of improving the efficiency of network transactions is to enable client side caching.

It is possible to add Cache Control headers to the content that we wish to make cacheable. Lotus Quickr includes these headers in the stylesheets that it uses by default, allowing the content to be cacheable at a client for 432,000 seconds (5 days)

We can use mod_headers in IBM HTTP server to add the same headers to images and JavaScript files by adding the following lines to the httpd.conf file:

LoadModule headers_module modules/mod_headers.so

Header add Cache-Control "public, max-age=432000, post-check=172000"

See this technote for more information on http response cache headers: http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21289550

Web Server TuningTuning (cont'd.)

I

In addition to gzip compression and cache headers we also tune the following parameters in our environment:

You can find detailed tuning information on IBM HTTP server at the following url:

http://publib.boulder.ibm.com/httpserv/ihsdiag/ihs_performance.html

Tuning (cont'd.)

Web Server Tuning

Network Tuning on Windows

Windows Server 2003 is designed to be largely self-tuning operating system. A standard, plain installation of the operating system yields reasonably good performance results for most purposes.

In some instances, specific server settings and parameters can be tuned to optimize the performance of the Windows server operating system even further.

We apply several tuning steps to our network parameters on the Windows OS to fully exploit the network bandwidth that is available.

You should optimise your network card settings for your particular network environment. Important parameters to consider include the speed and duplex settings of your network cards.

Consult the Lotus Quickr tuning Guide for information on basic network tuning steps for Windows.


There is also an excellent and indepth guide to tuning the Windows 2003 OS available as a red paper from this url: http://www.redbooks.ibm.com/abstracts/redp3943.html

Network Tuning on Linux

It is also important to consider the speed and duplex settings on Linux systems and ensure that they are compatible with your network environment as incorrect settings can cause a significant degradation in overall performance.

Consult the Lotus Quickr tuning Guide for information on basic network tuning steps for Linux.


There is an excellent tuning guide for Red Hat available as a red book at the following url: http://www.redbooks.ibm.com/abstracts/redp3861.html?Open

Monitoring and Ongoing Tuning

You should make use of monitoring tools to identify bottlenecks and areas for potential additional tuning in your environment.

You can use readily available tools like 'Perfmon' on windows and nmon on Linux to monitor resources and detect bottlenecks.

http://www.ibm.com/developerworks/aix/library/au-analyze_aix/

http://technet.microsoft.com/en-us/library/bb490957.aspx

You can monitor servlet threads, and other application data using PMI.

Carefully monitor the database system for potential bottlenecks.

There is no definitive 'one size fits all' tuning that can be applied.

Performance and Tuning are dependent on multiple factors and performance should be carefully monitored as the system scales and tuning adjusted based on the data collected.

It is recommended that you regularly schedule data collection from your monitoring tools and review the data collected.

Section Summary

Performance and Tuning considerations - What have we covered in this Section?

General Performance Recommendations.

Performance considerations relating to installation.

System specifications used in SVT and recommended Cluster configuration.

Performance considerations:

DB maintenance.

Search optimisation.

Tuning:

Portal/App server tuning.

DB tuning.

HTTP server Tuning.

OS and Network related tuning.

Usability Performance Tips.

Monitoring and ongoing Tuning.

What do We base Our Recommendations on?

Where do our recommendations come from?
(Based on various experiences in our labs and our internal deployments)

Statistics for Internal Sample Deployment

Synthetic site built for performance benchmarking

Model Topology: Single Server

Directory of 100,000 users

10,000 users have access to Lotus Quickr

512 teamspaces containing about 145,000 pieces of content

100,000 Documents

45,000 other content items (blog posts, wiki pages)

Database Size: 89.5 GB

Statistics for Internal Sample Deployment

Internal deployment for reliability testing

Run at 75% capacity for 5 days with acceptable response times

Approximates a 6 month old customer deployment (at start)

Model Topology: Single Server and Clustered Server

Directory of 10,000 users

100 Teamspaces containing about 304,000 pieces of content

100,000 Documents (average size: 500KB)

22,000 Wiki pages (average size: 41KB)

22,000 Blog entries (average size: 4KB)

160,000 List entries

Database size: 55 GB

Roughly .5M pieces of content at the end of a 5 day run

HTTP ServerDB2with FastT storageLotus Quickr Service for WebSphere Portal) ServersDeploymentManagerExample Deployment: IBM Internal Site

IBM Technology Adoption Program (TAP):

Number of Quickr Users = 18,500+

Number of Quickr Places = 6,000+

Total size Database = 430+ GB

IBM Lotus Greenhouse

Number of places: 4,499 J2EE places

Number of registered users: 15,227

Basic Troubleshooting

Basic Troubleshooting

The Troubleshooting process:

There is no magic solution, troubleshooting requires data collection and analysis performed in an iterative fashion.

What type of issue are we seeing? Is it a functional failure or performance related?

Known problems are documented in the form of individual technotes in the Support knowledge base at http://www-01.ibm.com/software/lotus/products/quickr/support/. As problems are discovered and resolved, the IBM Support team updates the knowledge base. By searching the knowledge base, you can quickly find workarounds or solutions to problems.

Some basic info to consider:

Is Lotus Quickr at the latest level, currently 8.1.1

CPU utilisation for Portal and for the other components.

Are the minimum H/W & S/W requirements met?

Has the system a remote DB? Were there any issues during DB transfer

What is the system configuration (Single server/Cluster) and what was the install/upgrade path.

What is the search index size, find the location of the index in the icm.properties file.

A good place to start is the SystemOut/SystemErr.logs.

Verify that the installed PK's match the recommended list.

You can find a list of the Portal PK's in SystemOut.log or Version.log

SystemOut.log and SystemErr.log are located in the \Quickr\PortalServer\log dir on a Single server or node 1 of a cluster and in the dir \Quickr\wp_profile\logs\WebSphere_Portal_name2 on node 2 in a cluster.

Basic Troubleshooting cont'd.

For database issues, examine the db2diag.log file from the DB2 server.

Use event monitoring and snapshot collection to gather data:

http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/r0005991.htm

http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/c0006003.htm

When was the last runstats/re-org performed on the database.

What is the total size of the DB, is there enough free space on the db server?

Some useful commands to retrieve configuration information from db2 include:

db2 "get db cfg for wpsdb" > dbcfg.txt

db2 "get dbm cfg" > dbmcfg.txt

db2set -all > db2set.txt

db2 "select * from syscat.bufferpools" > dbbuffpools.txt

Some useful queries that can be used to retrieve the amount of Documents in a db and the amount of webcontent are:

SELECT COUNT(*) FROM DB2USER.ICMSTJCRWSNODES WHERE CTID IN (select nodetypecomptypeid from db2user.icmstjcrnodetypes where nodetypename='clbDocument')

SELECT COUNT(*) FROM DB2USER.ICMSTJCRWSNODES WHERE CTID IN (select nodetypecomptypeid from db2user.icmstjcrnodetypes where nodetypename='webContent')


When troubleshooting a specific issue it can be helpful to increase the log file sizes and the backups of these files. This is especially true if applying tracing.

To increase the log file sizes on a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp


Now select Logging and Tracing from the resulting screen.

Now select JVM Logs from the resulting screen.

On this screen you need to change the maximum size of SystemOut and SystemErr to 50 MB and the Maximum Number of Historical Log Files to 300.

Select OK and then save the changes.

In certain circumstances it can be helpful to apply tracing to try to identify the issue.

If applying tracing you should minimise the activity on the system during this period.

To set tracing on a Cluster, login to the deployment manager admin console @ https://dmgrhostname.domain:9043/ibm/console/logon.jsp

Select Servers, Application Servers on the left hand side and then select WebSphere_Portal

Select Change Log Detail levels(under troubleshooting, near the bottom)

Select the runtime tab and set the trace string

Click apply at the bottom of the page, the change occurs without a restart.

Some common strings are shown on the next slide:

JCR Trace String

com.ibm.icm.jcr.NodeImpl=finest:

com.ibm.icm.jcr.WorkspaceImpl=finest:

com.ibm.icm.jcr.PropertyImpl=finest:

com.ibm.icm.jcr.query.*=finest:

com.ibm.icm.da.portable.common.sql.PPreparedStatement=finest

For Debugging Searches add

com.ibm.icm.ts.*=finest:

Wikis/Blogs add this trace on top of JCR

com.ibm.workplace.wcm.teamspace.*=finest:

com.ibm.workplace.wcm.services.repository.*=finest:

Team Calendar add this trace on top of JCR

com.ibm.workplace.team.calendar.*=all:

com.ibm.workplace.teamcalendar.*=all:

com.ibm.workplace.wcm.teamspace.*=all:

com.ibm.wps.portlets.Tcalendar.api.TCalendarAPI.*=finest:

com.ibm.wkplc.team.calendar.*=finest:


If there is a reproducible problem or failure, the process we have found most suitable for collecting focussed logs relating to the problem is:

Set up the trace and log file options

Login to Quickr

Navigate to right before the point where the problem occurs.

Record the start time from the trace log

Recreate the problem as normal.

Record the stop time from the trace log

Collect logs and examine in the period where the issue has occurred.

You can also collect DB2 snapshot and event monitor information as required.


For any OOM issues:

Verify that the correct WAS PK's are installed as recommended.

Verify the version of WAS JVM is correct: For 811 we tested with SR11.

Run \java\bin>java -version to get this information

Follow the process detailed here for OOM's: http://www-1.ibm.com/support/docview.wss?uid=swg21111364


Rebuilding Juru Indexes (Troubleshooting)

If we need to cleanly rebuild our JURU search indexes for any reason we use the following process:

Stop all servers.

Delete index from the search directory (the "1" directory within the Search directory)

On your DB server run the following commands:

delete from jcr.icmstjcrtspending

delete from jcr.icmstjcrtspathinfo

Restart your servers.

The indexes will start building again around 30 minutes after the start of Portal server if the default interval value is left unchanged in the icm.properties file.

You can monitor the SystemOut log to see the index activity.

The search directory will also keep increasing in size until indexing is complete. You can also run the query db2 select count(*) from jcr.icmstjcrtspathinfo during indexing, this value will keep increasing during the indexing build process.

We have found it optimal to backup our databases and search indexes in sync. This way if a database needs to be restored we can restore the associated index as well removing the need for a resynchronisation between the content and the indexes.

Tips and Tricks

Before making any major modifications to the system, take a full system backup including indexes and databases.

Backup any property files before modifying them.

When backing up your database it is best to backup your search indexes as well to keep the data in sync.

We have found significant space savings by compressing our database backups both on Windows and Linux. There is a consideration when doing this however as depending on the size of the backup/machine specifications it can take a number of hours to compress/decompress the backups.

It is helpful to 'tail' Logs when you are troubleshooting, on Linux this can be done with the tail command, On windows there are several alternative 'tail' programs that can be used.

Find WAS information run the script historyInfo.bat from the dir Quickr\AppServer\bin, this will generate a report called versionReport.html in the same directory with detailed information on the level and patches applied.

Find Portal information Run the script genVersionReport.bat from the directory: Quickr\PortalServer\bin This will create a detailed report called: versionReport.html in the same directory. You can create a detailed history report by running the script: genHistoryReport.bat from the \PortalServer\bin directory, this will generate a report called historyReport.html in the same directory.

List patches installed Check the version.log file, to review the history review the contents of the directory: Quickr\PortalServer\version\history

Section Summary

Basic Troubleshooting - What have we covered in this Section?

Basic Troubleshooting:

The process.

Items to consider.

Log Locations.

Database issues.

Tracing

Process for MustGather information for OOM's.

Tips and Tricks.

How we rebuild our search indexes.

Useful links/Resources.

Useful Links/Resources

IBM Lotus Quickr Version 8.1.1 Information Center @ http://publib.boulder.ibm.com/infocenter/lqkrhelp/v8r0/index.jsp

The hardware and software requirements for Lotus Quickr 8.1.1 can be found in detail @ http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg27009740

8.1.1: Readme for upgrading IBM Quickr 8.1 services for IBM WebSphere Portal -- Clustered Configuration @ http://www.ibm.com/support/docview.wss?rs=3264&uid=swg27013257

Required post-install fixes for Lotus Quickr 8.1.1 services for WebSphere Portal @ http://www-01.ibm.com/support/docview.wss?rs=3264&uid=swg21330316

Recommended 8.1.1 fixes @ http://www.ibm.com/eserver/support/fixes/fixcentral/swg/quickorder?brandid=2&productid=Lotus+Quickr&searchtype=recommended&vrmf=8.1.1

To view all technotes for Lotus Quickr go to the Lotus Quickr Support page @ http://www-306.ibm.com/software/lotus/products/quickr/support/

Lotus Quickr: Product documentation, white papers, Redbooks, and more - http://www.ibm.com/developerworks/lotus/documentation/quickr/

Lotus Quickr Support page - http://www-306.ibm.com/software/lotus/products/quickr/support/

Lotus Quickr product home page on ibm.com - http://www-306.ibm.com/software/lotus/products/quickr/

IBM DB2 Database for Linux, UNIX, and Windows Information Center - http://publib.boulder.ibm.com/infocenter/db2luw/v9//index.jsp

WebSphere Portal and Workplace Web Content Management product documentation - http://www.ibm.com/developerworks/websphere/zones/portal/proddoc.html

A comprehensive list of recommended, generally available (GA) fixes for IBM Lotus Quickr releases. @ http://www-01.ibm.com/support/docview.wss?rs=3264&context=SSVGKL&dc=DA400&uid=swg27010121&loc=en_US&cs=UTF-8&lang=en&rss=ct3264lotus

Portal Tuning Guide @ http://www-1.ibm.com/support/docview.wss?rs=688&uid=swg27008511

Lotus Quickr Tuning Guide @ http://www-1.ibm.com/support/docview.wss?rs=3264&uid=swg27010632

Diagnosing Performance Problems for WebSphere Portal 5.1 (though this document was written for WebSphere Portal 5.1, the lessons can apply to Lotus Quickr as well): http://www.ibm.com/support/docview.wss?uid=swg27007059.

IBM Corporation 2009. All Rights Reserved.The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBMs current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBMs sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

IBM, the IBM logo, DB2, DB2 Universal Database, Lotus, Domino, Quickr, Redbooks and WebSphere are trademarks of International Business Machines Corporation in the United States, other countries, or both.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Machine FunctionOperating SystemSoftwareMachine Specifications(All Systems are IBM Blades connected via fiber to a high speed disk array )

Deployment managerRed Hat Enterprise Linux ES release 4/Windows 2003 EEWAS ND 6.0.2.172 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

Quickr Server - 2 nodes

Red Hat Enterprise Linux ES release 4/Windows 2003 EEQuickr 8.1.1 / WAS 6.0.2.17 / WPS 6.0.1.1 / JRE 1.4.2 sr112 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

DB2 serverRed Hat Enterprise Linux ES release 4/Windows 2003 EEIBM DB2 Universal Database Enterprise Server Edition 9.1 FP5 - DB2 JDBC Universal Driver (Type 4)2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

LDAP serverRed Hat Enterprise Linux ES release 4/Windows 2003 EEIBM Tivoli Directory Server v6.1 server2 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

WebServerRed Hat Enterprise Linux ES release 4/Windows 2003 EEIBM HTTP Server 6.0.2.29/2.0.47.12 Dual-Core(Intel Xeon) CPUs @2.33GHz - 4 GB Ram

Click to edit the title text format

2008 IBM Corporation



IBM Software Group | Lotus software







Express Install.Only suitable for evaluation or very light departmental usage.

Advanced Single Server.Use this option when planning to extend to a cluster at some point in the future.

Advanced Enterprise Cluster.This is the recommended configuration for a production level environment.

Community

JCR

Release

Everything else


Initial and Maximum Heap Size (MBytes)Max=1408Max=1408

Considerations for your DB server

Using a remote database can have performance benefits.

Under higher loads, the application becomes database intensive.

This database activity can increase CPU utilisation and disk I/O time and reduce system capacity.

Separating the database from the server that Quickr is running on increases capacity.

Ensure there is adequate disk space for the database transfer, if you are transferring a large database then you will need to consider the amount of disk space that needs to be free.

Ensure there is adequate disk space for the database to grow in the future based on usage predictions.

The database server needs to have a powerful disk system. This should ideally be a dedicated high speed disk array such as a fibre connected SAN such as FAStT or a high speed internal disk array.

In our experience Linux based database servers performed better in terms of disk I/O.

WindowsLinux

db2 alter bufferpool ICMLSFREQBP4 size 12000db2 alter bufferpool ICMLSFREQBP4 size 5000

db2 alter bufferpool ICMLSVOLATILEBP4 size 72000db2 alter bufferpool ICMLSVOLATILEBP4 size 30000

db2 alter bufferpool ICMLSMAINBP32 size 30000db2 alter bufferpool ICMLSMAINBP32 size 26000