daniel reimann | panagenda - client_server... · event_correlation_pool_size. •...

$: Daniel Reimann | panagenda - Client_Server... · EVENT_CORRELATION_POOL_SIZE. • Monitor.Last.INDEX ALL.Warning(High)Text = Error updating view '#4538' in mail\nameabc.nsf: The single$
Daniel Reimann | panagendawww.panagenda.com

Efficiency describes the extent to which time or effortis well used for an intended task or purpose.

Client Server and Application Monitoring and Optimization done right 2

Agenda

Coming up next …

Who am I? … and about panagenda

Laying the basics of what is actually possible – or:• What Admins and IT departments have to cope with

Deep Diving …• The 30 most important server statistics (out of ~2.000)

• … and Clients?

• … and Groups?

• … and Databases?


Introduction


Daniel Reimann

Head of Technical Account Management - panagenda

Lotus Notes / Domino since 1999

Consultant at several IBM business partners• Administration and Development• Also long-term for well-known big companies

TAM @panagenda since 2011 with core competence on Lotus Notes Client Management,Lotus Notes / Domino Infrastructure-Analysis and -Optimization

I enjoy working with many great companies in many different countries (I travel *a lot*)


About panagenda

We network symbiotic relationships with our customers and partners for ongoing joint win-win

HQ: Vienna/AT, offices in Heppenheim near Frankfurt/DE, Boston/USADevelopment of standard products> 4 million licenses in over 70 countries

IBM Lotus NotesClient ManagementMarvelClient :: „99%“ manageability

(not „just“ IBM Lotus Domino)Server Analytics, Monitoring & ReportingGreenLight :: realtime, longterm, smart

Analyze Groups, Certifiers and ACLsGroupExplorer :: better transparency, security & automation

plus: NameChanger (Name changes), DatabaseExplorer (Design Analysis), Notes2Web (Web transformation)


Agenda

Coming up next …





• … and Groups?



What Admins and IT departments have to cope with

• Above all: Lack of knowledge (apologies)• Mostly because of overstress No time (anymore) for the inner workings of clients, servers, and systems Growing complexity of single systems Growing number of systems „Laying the egg“ = yes;

(Proactive) „Nurturing“ = no.• Unknown sources of knowledge

• Lack of time• If you don't take the time to do things right

you’ll need the time to do them over

• „Wrong“ &| missing toolingGrown environments: large servers are fundamentally different

from small ones; new ones (8) from old ones (< 8)!

Newborn bear 3 month oldbear, without fur

Full-grown teddywith thick fur

Development stages of teddybears

Presenter

Presentation Notes

�Falsche Werkzeuge insbesondere im Sinne von „nicht ausreichend“ oder gar schlimmer „einen nicht ausreichenden bis falschen Eindruck vermittelnd“


What Admins and IT departments have to cope with

Systemic interactions / dependencies in Lotus Notes / Domino

Servers

DatabasesClients

Hardware (CPU, Memory)Data storageNetwork connectionConfigurationDatabases, tasks, mail traffic…

ODSSizeReader fieldsDesign# & Size of documents…

HardwareData storage

NW connectionConfiguration

Databases…

Across all:GeographiesNetwork (bandwidth, structure)Online/OfflineClustering/Loadbalancing…

Presenter

Presentation Notes

An dieser Stelle sehen wir uns zuerst einmal nur den Lotus Domino server an …


Public NAB– Servers

– Clusters

– People/Groups

– Directory

– Messaging

– Replication

– Policies

– Web Configuration

Lotus Domino „out of the box“ tooling



Public NAB (8) Log.nsf

– Miscellaneous(!!)

– Mail

– Replication

– (Database) Usage

– Passthru Connections



Public NAB (8) Log.nsf (5) Admin Client

– Monitoring

Tip 1: Enable Health-Monitoring in Admin Preferences

Tip 2: Disable „Refresh serverbookmarks“



Public NAB (8) Log.nsf (5) Admin Client

– Monitoring (1)

– Analysis(~15)(ACL,Catalog,AdminP, ...)

– Statistics (1 or ~1.200)

– Activity Trends („1“)

– Messaging („1“)

– Replication („1“)



Public NAB (8) Log.nsf (5) Admin Client (20) Events (1) & DDM …

– Probes

– Filters

– Collection Hierarchy

– Event Handlers

– Event Generators



Public NAB (8) Log.nsf (5) Admin Client (20) Events & DDM (6) Monitoring Results (statrep)

– Alarms

– Events

– Statistic Reports



42Although 42 is

„the answer to life,the universe and everything“

(according to the Hitchhikers Guide to the Galaxy)that doesn‘t help much

for LN/D Monitoring & Analysis

Tip 3: In case you don‘t know the Hitchhikers Guide to the Galaxy from Douglas Adams Must Read

Public NAB (8) Log.nsf (5) Admin Client (20) Events & DDM (6) Monitoring Results (statrep)

(3)

8 + 5 + 20 + 6 + 3 = 42

That‘s at least 42 views / areas, one should monitor ...


Making more of what you already have

• Many companies don‘t even use what‘s in the box already …• (As said earlier): Realtime Server Monitoring with Health Monitoring• DDM – Domino Domain Monitoring (sometimes a bit too much, but then again much better than nothing!)• Frequent reviews of Groups• Frequent checking of the most

important server stats(more of that later)

• Look through Lotuspherepresentations

• …• Investigate Usage-views in log.nsf;

for example …


A sample analysis of usage information from log.nsf

• (that you can do yourself easily)Copy/Paste in Excel Sort data by i.e.Transactions


Possibilities are endless (unfortunately, time is not)

• In almost all of the beforementioned areas one can(and should) „dig deeper“

• Unfortunately, digging deeper requires (time-consuming) correlation of data, e.g. …• Connection documents and log.nsf (db usage): How much Mail- and/or Replication traffic is there

between which servers?• Clients and log.nsf - database usage: Which users cause what load from where?

• Database details from clients and servers: Who has replicas of databases s/he no longer has access

to? Who has (unencrypted) replicas of critical databases?

• Network compression between servers and clients

• A lot of the data is either already there or(relatively ;-)) easy to get a hold of

• Correlation pays back (repeatedly) …


A picture says a thousand words …

• Topological visualization of Mail- & Replication-Traffic between Servers



75%of your IBM Notes Clients use port compression(35,409 of 47,212 clients)

= 1.000 Clients

87%of your IBM Lotus Domino servers use port compression(33 of 38 servers)

= 1 Server

Presenter

Presentation Notes

The only statistics you can trust are those you falsified yourself



2.30 3.30 1.65024

current setup no port compression full port compression

saved (GByte)transfered (GByte)

● Network transfer volume per day: 3.3 Gbyte

● Current settings: 75% configured „correctly“ ~1 GByte / 30% saved

● Applying port compression to all your servers and clients could save you an additional ~0.65 GByte every day which is an additional 28% reduction / absolute 50% reduction of traffic

Presenter

Presentation Notes

The only statistics you can trust are those you falsified yourself


Agenda

Coming up next …





• … and Groups?



Before we look at the 30 most important server statistics …

• Difficult – if not impossible – to test in the lab• Start with the obvious / easy things• Note down current settings before changing them• Think in possible interdependencies• „Too much good“ can actually harm performance

(or lead to „Out of Memory“)• Don‘t change (too) many things at once• Unless it‘s absolutely necessary / so „documented“

• Watch your servers for some (sense making) time after making changes• Check whether/that your servers are doing better

• „Google“• Think along/ahead• Have the heart to try• This is just the beginning – stay curious!


And another preliminary note (last one(s), promised ;-))

• Many of the following statistics cannot be grasped with a ‚single‘ „sh sta“, but require analysis„over time“• Otherwise you won‘t know whether you‘re looking at a permant / recurring / onetime / sometime problem• Otherwise you won‘t know whether changes actually improved things (or made things worse)• A picture says a thousand words …

• Admin Client can be usedas a starting point …(unfortunately, it is very limited)


ViewRebuildDir & Disk optimization(s)

• Most important of all: free disk space & disk performance(„30%“ to prevent fragmentation)

• Seperate, dedicated disks for …– Translog

– Data– If possible, own disk for page file/OS– „ViewRebuildDir“=…

view indexing on its own disk– From 8.5.3. on where necessary/wanted

.ft-directories on own disk– DAOS („cheap“)


Platform.LogicalDisk.*• Platform.LogicalDisk.1.AssignedName = D

Platform.LogicalDisk.1.AvgQueueLen = 0Platform.LogicalDisk.1.AvgQueueLen.Avg = 0,01Platform.LogicalDisk.1.AvgQueueLen.Peak = 1,01Platform.LogicalDisk.1.BytesReadPerSec = 0Platform.LogicalDisk.1.BytesWrittenPerSec = 10.172,49Platform.LogicalDisk.1.PctUtil = 0,22Platform.LogicalDisk.1.PctUtil.Avg = 0,86Platform.LogicalDisk.1.PctUtil.Peak = 101,07Platform.LogicalDisk.1.ReadsPerSec = 0Platform.LogicalDisk.1.WritesPerSec = 2,07

Platform.LogicalDisk.2.AssignedName = CPlatform.LogicalDisk.2.AvgQueueLen = 0,01Platform.LogicalDisk.2.AvgQueueLen.Avg = 0,73Platform.LogicalDisk.2.AvgQueueLen.Peak = 34,74Platform.LogicalDisk.2.BytesReadPerSec = 17.272,75Platform.LogicalDisk.2.BytesWrittenPerSec = 63.697,52Platform.LogicalDisk.2.PctUtil = 1,11Platform.LogicalDisk.2.PctUtil.Avg = 72,8Platform.LogicalDisk.2.PctUtil.Peak = 3.473,81Platform.LogicalDisk.2.ReadsPerSec = 2,58Platform.LogicalDisk.2.WritesPerSec = 7,3

InterpretationGOOD < 2% < AvgQueueLen > 5% > BAD (1-100% = 0,01 – 1,0!)GOOD = PctUtil < 80% (1-100% = 1-100)NOTE: may need to divide by # of spindles SAN/NAS

SolutionVarious parameters (bufferpool, cache, namelookup) and OS / Disk Tuning


Platform.LogicalDisk.#.PctUtil


Server.Availability• Shows how available = „ready to respond“ a Server is (in %)• < 30% means trouble (or loadbalancing);

IF the Availability Index is correct in the first place … (Only!) if the server is well busy: „sh ai“ on server console;

results in recommendation on how to tune ini:SERVER_TRANSINFO_RANGE

•From notes 8.5 and up, you are advised to set:– notes.ini: Server_MinPossibleTransTime=1500– notes.ini: Server_MaxPossibleTransTime=20000000

Important:Delete loadmon.ncfafter server shutdownin order to delete oldvalues


Keep an eye on Monitor.* Warnings; Examples

• Monitor.Last.ADMIN PROCESS.Warning(High)Text = Disk space statisticscould not be found on Servername/Cert.

• Monitor.Last.EVENT MONITOR.Warning(High)Text = Event: Error adding eventdocument to Domino Domain Monitoring: Event correlation cache is full. Youcan increase its size via the NOTES.INI settingEVENT_CORRELATION_POOL_SIZE.

• Monitor.Last.INDEX ALL.Warning(High)Text = Error updating view '#4538' inmail\nameabc.nsf: The single copy template associated with this databasecannot be located.

• Monitor.Last.SMTP SERVER.FailureText = SMTP Server: Initializationfailure: Message Queue name already in use.

• Monitor.Last.STATISTICS.Warning(High)Text = Unable to update activitydocument in log database for mail\namexyz.nsf: In Datenbank kann nichtgeschrieben werden, da die Datenbank die erlaubte Größeüberschreiten würde.


Server.Sessions.Dropped• Tells you how many sessions have been

‚dropped‘ since last server restart• Happens when• issuing a serverside „Drop all“• Pressing Ctrl+Break on clients („frustration-

meter“)• Server.Sessions.Dropped = 25407• 18/6 – 18/10 = 4*30 = 120 days• 25407 / 120 = 211

sessions dropped per day• Should be further correlated with peak # of

users

„different“Problem

„Drop all“


Mail.Mailbox.*

• Mail.Mailbox.AccessConflicts/Mail.Mailbox.Accesses) x 100• Must be < 2, otherwise: add another Mailbox

(benefit increase decreases above 4-5 mailboxes)

• Example:• Mail.Mailbox.AccessConflicts = 1636

Mail.Mailbox.Accesses = 189864

• = 0,86 = ok


Update.PendingList• Update.PendingList = number of

views waiting to be updated• If

Update.PendingList„is often“ > 0, then …

• Notes.ini:Update_Fulltext_Thread=1FTUPDATE_IDLE_TIME=4

Background:• If you have many databases/apps …• … and a busy update task

– Full text index could be the reason for slowing down / “blocking” view indexing

• Separate FTI and view updates– FTI then runs in its own Memory Thread

• Improves performance• Update_Fulltext_Thread=1

Speaking of Fulltext-Indexing:You can isolate the FTI thread from the limited Domino update pool:

ftg_use_sys_memory=1FTI thread then gets memory from OS pool;

relieves Domino system memory


Database.Database.BufferPool.*

• Database.Database.BufferPool.PerCentReadsInBuffer = 78,96

• BAD < 90% < PercentReadsInBuffer < 98% < PERFECT(99.9% is bad, too!)– Typically leads to too many requests being written to disk– Server needs a larger BufferPool

• Solution: notes.ini NSF_Buffer_Pool_Size_MB=n (in MB)─ Default: 512 MB


Database.DbCache.*

• Database.DbCache.CurrentEntries = 1647Database.DbCache.HighWaterMark = 1691Database.DbCache.MaxEntries = 1536Database.DbCache.OvercrowdingRejections = 0

• GOOD = HighWaterMark < MaxEntriesGOOD = 0 OvercrowdingRejections

• Solution:– notes.ini NSF_DbCache_MaxEntries = n

• Default: NSF_BUFFER Pool size x 3


Replica.Cluster.*

• Replica.Cluster.FailedReplica.Cluster.SecondsOnQueueReplica.Cluster.WorkQueueDepth

• PERFECT < 10 < SecondsOnQueue > 15 > BADPERFECT < 10 < WorkQueueDepth > 15 > BAD

• Solution:– Add more cluster replicators– Optimize cluster load

(e.g. “manually” balanceusers across cluster if notload-balance)


Server.Trans.PerMinute

• Server.Trans.PerMinute=956Server.Users = 26 956/26=36,7

• HEAVY < 30 < Trans.PerMinute (per User) > 10 > LIGHT

• Solution:– Identify users causing load (db usage view!)


Database.NAMELookupCache*

• Database.NAMELookupCacheCacheSize = 2.513.328Database.NAMELookupCacheHits = 24.628.339Database.NAMELookupCacheMisses = 48.160.502

• IMPORTANT: NoHitHits! Cache too small or too large(!)• Miss > Hits: „Doublecheck“

ini:NLCache_Size=16000000


Server.ConcurrentTasks*

• Server.ConcurrentTasksServer.ConcurrentTasks.Waiting

• Waiting should be ZERO (0)

• Solution:─ Server_Pool_Tasks = n (e.g. 80)─ Server_Max_Concurrent_Trans = m

(e.g. Server_Pool_Tasks * # Ports)


Platform.PagingFile.Total.*• Platform.PagingFile.Total.PctUtil = 0,28

Platform.PagingFile.Total.PctUtil.Avg = 0,14Platform.PagingFile.Total.PctUtil.Peak = 0,8

• OK < 0% < PctUtil.Avg > 10% > BAD

• OS Level tuning, Check Memory

• Note: If “sh sta” doesn’t show Platform.* stats Admin-Help


Agenda

Coming up next …





• … and Groups?



Sponsor Break – Sneak Peek during Social Evening

http://panagenda.com/giftoftransparency

• Efficient Client-Analysis is impossible without additional tooling

• FREE 4 weeks license of panagenda GreenLight – our servermonitoring and reporting solution – includes Database Analyzer for1 year for one of your servers

• FREE one year license of panagenda MarvelClient Analyze• The results speak for themselves on „just“ the clientside• The results can also be used together with GreenLight

• For groups and databases, wie also have GroupExplorer andDatabaseExplorer• Whether we may help you is up to you


Timeout

Spending 45 minuteson Performance Improvements

can be compared toa walk on the tip of the iceberg –

we have worked ona MANY more business cases

and solved a MANY more problemsthan those mentioned just now.

If your problem was not mentioned in this session –be it a Client, Server, Design, Admin

or other challenge:we would love to hear from you.


Thank you for listening – Questions? Answers!

Q&A


Contact us

panagenda GmbH

Doblhoffgasse 7 / 6a :: 1010 Vienna :: AustriaWeb: http://www.panagenda.com

Email: [email protected]: +43 1 89 012 89 – 15

Your contact in NorwayEVRY Consulting AS – Roger Johannessen


Ressources / Links

• Daniel Nashed, Nash!Com• LS08: BP112• LS11: BP102, BP110, BP118• LS12: BP110, BP121, ID112, ID114• Windows Indexing: http://bit.ly/ACzO6Z• „The internet“ – google „Domino performance ibm“;

great IBM Whitepapers and articles, some very good site out there

http://bit.ly/ACzO6Z�

daniel reimann | panagenda - client_server... · event_correlation_pool_size. •...

Documents