domino performance best practices on iseries - mitec€¦ · domino performance best practices on...

Domino Performance Best Practices on

iSeriesKim Greene

[email protected]

Copyright (c) 2006 Kim Greene Consulting, Inc.. All rights reserved worldwide.

2Copyright © 2006 Kim Greene Consulting, Inc. All Rights Reserved Worldwide

Introduction - Kim Greene

President of Kim Greene ConsultingPresident of Kim Greene ConsultingServices offered include:Services offered include:

ƒƒSystem and application tuningSystem and application tuningƒƒEnterprise integrationEnterprise integrationƒƒAdministration and troubleshootingAdministration and troubleshootingƒƒDomino application developmentDomino application developmentƒƒCustomized education and trainingCustomized education and trainingƒƒWorkplace installation and setupWorkplace installation and setupƒƒWorkplace performance tuningWorkplace performance tuningƒƒTechnical writer for Technical writer for eServereServer Magazine and iSeries Experts Magazine and iSeries Experts JournalJournal

Member of Penumbra GroupMember of Penumbra Groupƒƒwww.penumbra.orgwww.penumbra.org


What We'll Cover...

Mail file performanceMail file performanceƒƒSize of mail filesSize of mail filesƒƒImpact from number of documentsImpact from number of documentsƒƒFull text indexes and searchingFull text indexes and searching


What We'll Cover...

Server tuningServer tuningDomino memory managementDomino memory management

ooNSF buffer poolNSF buffer poolooDBcacheDBcache

When tasks runWhen tasks runAre you wasting memory?Are you wasting memory?

ooSettings for *INTERACT and *SPOOLSettings for *INTERACT and *SPOOL


What We'll Cover...

Application tuningApplication tuningLotus Enterprise Integrator (LEI)Lotus Enterprise Integrator (LEI)

ooImportance of correct data typesImportance of correct data types

DDMDDMooGet control over agent performanceGet control over agent performance

Run prioritiesRun prioritiesooImportance of considering other workloadsImportance of considering other workloads


What We'll Cover...

Application deployment considerationsApplication deployment considerationsOn mail server?On mail server?Separate application server?Separate application server?

vvDomino 7 performanceDomino 7 performanceDifferentiating Differentiating benchmarketingbenchmarketing with with realreal--lifelife


Mail Performance


Impact of Number of Documents

Number of documents has greater impact Number of documents has greater impact as file size growsas file size grows

0

2

4

6

8

10

12

14

16

100 MB 700 MB

1,024 docs4,096 docs7,196 docs28,672 docs


File Those Documents!

Inbox is most expensive container in a Inbox is most expensive container in a mail filemail fileƒƒUpdated each time user opens mailUpdated each time user opens mailƒƒUpdated each time user clicks Refresh Updated each time user clicks Refresh buttonbutton

Reducing number of documents in Inbox Reducing number of documents in Inbox reduces:reduces:ƒƒCPU utilizationCPU utilizationƒƒMain storage required to update the viewMain storage required to update the view


File Those Documents!

Reducing number of documents stored Reducing number of documents stored in Inboxin Inbox

°Mail file open performance improved 50%°Steady state performance improved 12%

0

5

10

15

20

25

30

25% in Inbox 100% in Inbox

OpenSteady State


Mail File Best Practices

For optimal performanceFor optimal performanceƒƒReduce overall document count in Reduce overall document count in the filethe file

••Up to 2X CPU reductionUp to 2X CPU reductionƒƒKeep number of documents stored in Keep number of documents stored in Inbox to a minimumInbox to a minimum


Full-Text Indexing and Searches

Full text indexing is cheapFull text indexing is cheapƒƒCPU usage increase < .5%CPU usage increase < .5%ƒƒBut consumes disk space!But consumes disk space!

Searches are expensiveSearches are expensiveƒƒ20% extra CPU20% extra CPU

Moral of the storyMoral of the storyƒƒCreate permanent indexes if using FTI functionCreate permanent indexes if using FTI function

FT_FLY_INDEX_OffFT_FLY_INDEX_Off=1=1ƒƒPrevents on the fly full text indexingPrevents on the fly full text indexing


Domino Memory Management



How Domino divides main storageHow Domino divides main storageƒƒDomino has it’s own memory pools/buffersDomino has it’s own memory pools/buffersƒƒMemory subpools are dynamically adjusted by Memory subpools are dynamically adjusted by DominoDomino

NSF Buffer

DB Cache

Apps Users



NSF Buffer Pool NSF Buffer Pool –– what it doeswhat it does

NSF Buffer PoolNIF Disk



NSF Buffer Pool NSF Buffer Pool –– how size is sethow size is setƒDomino R5

ƒSet to 300 MB by default on iSeriesƒDomino 6

ƒSet to 748 MB by default on iSeriesƒDomino 7

ƒSet to 1515 MB by default on iSeries



NSF Buffer Pool: NSF Buffer Pool: Should you keep the default size?Should you keep the default size?ƒƒRecommend setting based on actual server Recommend setting based on actual server needsneeds

ƒƒUseUse Show Stat DatabaseShow Stat Database command to view command to view statisticsstatistics



Domino database statisticsDomino database statisticsDatabase.Database.BufferPool.Maximum.MegabytesDatabase.Database.BufferPool.Peak.MegabytesDatabase.Database.BufferPool.PercentReadsInBuffer

Want >= 95%

Database.DbCache.CurrentEntriesNumber of entries in the database cache

Database.DbCache.HighWaterMarkPeak number of entries in db cache since Domino server was started

Database.DbCache.MaxEntriesNumber of databases that can be cached

Database.DbCache.OvercrowdingRejections



Customer example Customer example –– looking goodlooking good

Database.Database.BufferPool.Maximum.Megabytes = 710Database.Database.BufferPool.Peak.Megabytes = 708Database.Database.BufferPool.PercentReadsInBuffer = 98.12Database.DbCache.CurrentEntries = 71Database.DbCache.HighWaterMark = 256Database.DbCache.MaxEntries = 300Database.DbCache.OvercrowdingRejections = 0



Customer example Customer example –– looking very bad looking very bad Database.Database.BufferPool.Maximum.Megabytes = 170Database.Database.BufferPool.Peak.Megabytes = 168Database.Database.BufferPool.PercentReadsInBuffer = 95.44Database.DbCache.CurrentEntries = 510Database.DbCache.MaxEntries = 510Database.DbCache.OvercrowdingRejections = 8385

Huge problem!

Options:Increase NSF buffer pool sizeSet NSF_DbCache_MaxEntries



NSF_Buffer_Pool_Size_MBNSF_Buffer_Pool_Size_MBƒƒ What to monitor when fine tuningWhat to monitor when fine tuning

1. Non-database paging faulting = < 100 faults/processor2. Database.BufferPool.PercentReadsInBuffer >= 95%3. Database cache size

° Database.DbCache.MaxEntries = 3 X BufferPool



Two key notes.ini parameters affect Two key notes.ini parameters affect Domino memory managementDomino memory management

ƒƒNSF_Buffer_Pool_Size_MBNSF_Buffer_Pool_Size_MBƒƒPercentAvailSysResources PercentAvailSysResources

Other Domino memory buffers are Other Domino memory buffers are allocated based on these parametersallocated based on these parameters



PercentAvailSysResourcesPercentAvailSysResourcesƒƒIntroduced in R5.0.4Introduced in R5.0.4ƒƒControls % of memory resources Controls % of memory resources available to Domino serveravailable to Domino server

oValid values = 2% to 100%o Memory available depends on where Domino server is executing

ƒƒAllows dynamic adjustment of buffer Allows dynamic adjustment of buffer sizessizes



NSF_Buffer_Pool_Size_MBNSF_Buffer_Pool_Size_MBƒƒStatic setting of NSF buffer poolStatic setting of NSF buffer poolƒƒUse in combination with Use in combination with PercentAvailSysResourcesPercentAvailSysResources°PercentAvailSysResources won’t override NSF_Buffer_Pool_Size_MB setting


Know When Tasks Run


Know When Tasks Run

• Default Schedule of Domino TasksServerTasksAt1=Catalog,DesignServerTasksAt2=UpdAllServerTasksAt3=Object Info -Full o Only needed if using shared mailServerTasksAt5=Statlog Program

documents• Server tasks can be alternately

scheduled via program documents


Know When Tasks Run

• Program documentsShow Schedule will show the schedule for all program documents

• What is your Backup schedule?

• What other applications run on the system?

• What runs on the weekend?


Are You Wasting Memory?


Memory pool settings

• Check minimum size of memory pools*INTERACT*SPOOL

• Use ‘WRKSHRPOOL’PF11 Display Tuning Data


WRKSHRPOOL


iSeries Navigator

• Work Management -> Memory Pools -> Active pools

• Right click pool -> select ‘Properties’• Tuning tab


Are you Wasting Memory?

• Pool size minimums and maximums are used by automatic performance adjustor (QPFRADJ)

Adjusting minimums allows memory to be better utilized


Application Tuning


LEI

• Think data types aren’t important?Think again!!

• Does this message look familiar?*Replication Conflict* Action: Update at Connector B; Mismatch in field ‘QTYSHP’; Record Key values: WHRLW3=“G956105388117”


LEI

• What happens during replication?

78.894.050

1142.7561142.756

11.000

3.63.600

7070.000

NotesDB2

Replication conflict



OK, skippedReplication conflict


LEI

• How do we fix this?Change the field types to matchOriginal field typeso DB2

PackedField length 11,3

oNotesType = Number Number format = DecimalDecimal places = varying

oNotesType = NumberNumber format = DecimalDecimal places = 3 fixed

New field typeso DB2

PackedField length 11,3


LEI

• Now what do the data types look like?

78.80094.050

1142.7561142.756

1.0001.000

3.6003.600

70.00070.000

NotesDB2

OK, skipped

OK, skipped

OK, skipped

OK, skippedReplication conflict


LEI

• End result?Original replication timeo 1.5 hours

New replication timeo 12 minutes

* Number of records replicated = ~200,000


DDM

• Domino Domain Monitoring (DDM)New Domino 7 server monitoring featureo Enabled in Monitoring Configuration database

(events4.nsf)o Feature-oriented view of domain status in

DDM.nsfExtends event monitoring in D5 & D6Many different probes and events


DDM Overview

Monitoring Configuration(events4.nsf)

Domino Domain Monitor(DDM.nsf)

Used for all Domino domain monitoring configuration

Domino domain monitoring probesgenerate Event report documents that get consolidated and reported into the new DDM database


DDM

• Use probes to monitor performanceProbe type = Application CodeProbe subtypes

o Agents behind scheduleo Agents evaluated by CPU usageo Agents evaluated by memory usageo Long running agents

o Processes to probe = AMGR or HTTP


Where to Place Applications


Mixing Applications and Mail

• Where do you deploy applications?Mail serverApplication server

• Understand impact of updatesUpdate task works off queueFIFOCan get erratic performance


Domino 7 Performance



• Domino 7 scalabilityIncrease in number of users supported in a DPAR

o Domino 6 = 10,000 userso Domino 7 = 18,000 users

Domino web access (DWA) shows increased scalability

o 24% reduction in CPUo Up to 15,000 DWA users in a single DPAR



• New statistics availableUpdate tasko Show length of queues (pending and immediate)

Number of requests pendingNumber of requests in each queue

o Requests discarded because they were duplicates

o Date and time next immediate request will be processed



• New statistics availableFull text indexingo Number of unprocessed FT requests in FT queueo Number of requests discarded from FT queue

because they are dups of immediate requestso Maximum length queue has been since server

starto Total number of FT update requests performed


References


Domino Performance Redbook

Domino for iSeries Sizing and Performance Domino for iSeries Sizing and Performance Tuning on the IBM eServer iSeries Server, SG24Tuning on the IBM eServer iSeries Server, SG24--51625162--0101

ƒƒwww.ibm.com/redbooks/redbooks/pdf/sg245162.pdfwww.ibm.com/redbooks/redbooks/pdf/sg245162.pdf

Domino 6 for iSeries Best Practices Guide, SG24Domino 6 for iSeries Best Practices Guide, SG24--69376937

ƒƒwww.redbooks.ibm.com/redpieces/pdfs/sg246937.pdfwww.redbooks.ibm.com/redpieces/pdfs/sg246937.pdf


Domino Performance Redpapers

Sizing LargeSizing Large--Scale Domino Workloads on iSeriesScale Domino Workloads on iSeriesƒƒ publibpublib--b.boulder.ibm.com/Redbooks.nsf/RedpaperAbstracts/redp3802.html?Ob.boulder.ibm.com/Redbooks.nsf/RedpaperAbstracts/redp3802.html?Openpen

IBM Lotus Domino for iSeries Performance and IBM Lotus Domino for iSeries Performance and Tuning, REDP3843Tuning, REDP3843

ƒƒwww.redbooks.ibm.com/redpapers/pdfs/redp2843.pdfwww.redbooks.ibm.com/redpapers/pdfs/redp2843.pdf

iSeries Performance Management Resources iSeries Performance Management Resources LibraryLibrary

ƒƒwwwwww--1.ibm.com/servers/eserver/iseries/perfmgmt/resource.htm1.ibm.com/servers/eserver/iseries/perfmgmt/resource.htm


References

iSeries Domino PerformanceiSeries Domino Performanceƒƒhttp://www.ibm.com/servers/enable/site/domino/perform.htmlhttp://www.ibm.com/servers/enable/site/domino/perform.html

Performance ZonePerformance Zoneƒƒwww.lotus.com/performancewww.lotus.com/performance

Lotus Developer DomainLotus Developer Domainƒƒwww.lotus.com/lddwww.lotus.com/ldd

Ask Professor INIAsk Professor INIƒƒwww.lotus.com/ldd/today.nsf/profini?OpenViewwww.lotus.com/ldd/today.nsf/profini?OpenView

IBM Workload EstimatorIBM Workload Estimatorƒƒwwwwww--912.ibm.com/servlet/EstimatorServlet912.ibm.com/servlet/EstimatorServlet


Your Turn!

Questions?