Strength. Strategy. Stability.

Progress Performance Progress Performance Monitoring and TuningMonitoring and Tuning

Dan ForemanDan Foreman

Progress ExpertProgress Expert

BravePointBravePoint

BravePoint 2011 2

BravePoint 2011

3

Introduction – Dan ForemanIntroduction – Dan ForemanProgress User since 1984Progress User since 1984

Author of:Author of:Progress Performance Tuning Guide

Progress DB Administration Guide

Progress Virtual System Tables

ProMonitor

Pro Dump&Load

And other miscellaneous stuff

Introduction - AudienceIntroduction - Audience

Progress Version?Progress Version?V8

V9

V10?

DB Operating System: *ix or WindowsDB Operating System: *ix or Windows

Largest Single DatabaseLargest Single DatabaseOne of BravePoint’s customer’s has a Six Terabyte database

BravePoint 2011

4

Monitoring CategoriesMonitoring Categories

Progress DatabasesProgress Databases

Progress ClientsProgress ClientsCharacter

Graphical

AppServers & WebSpeed

SQL ClientsSQL Clients

Operating SystemOperating System

Application – beyond the scope of Application – beyond the scope of this presentationthis presentation

BravePoint 2011

5

Monitoring - DatabaseMonitoring - Database

GrowthGrowth

I/OI/O

Record LockingRecord Locking

After ImagingAfter Imaging

Before ImagingBefore Imaging

Resource ContentionResource ContentionLatches

Semaphores

BravePoint 2011

6

Monitoring - DatabaseMonitoring - Database

Database HealthDatabase HealthLog File

Blocks & Chains

Pointers (Index Data)

Database StructureDatabase Structure

Database “Demographics”Database “Demographics”

ReplicationReplication

BackupsBackups

BravePoint 2011

7

Monitoring ToolsMonitoring Tools

promonpromon

Virtual System TablesVirtual System Tables

Other Progress utilitiesOther Progress utilities

3d Party3d PartyProMonitor (BravePoint)

OpenEdge Management (Progress)

OtherOther

BravePoint 2011

8

Monitoring Tools - promonMonitoring Tools - promon

Old faithfulOld faithful

Character interfaceCharacter interface

Doesn’t record any historyDoesn’t record any history

No AlertingNo Alerting

Somewhat poorly documentedSomewhat poorly documentedSee Dan Foreman’s Performance Tuning Guide

Takes a ‘snapshop’ of DB Shared Takes a ‘snapshop’ of DB Shared MemoryMemory

BravePoint 2011

9

Monitoring Tools – VSTsMonitoring Tools – VSTs

Access Shared Memory using System Access Shared Memory using System TablesTables

Doesn’t have everything Doesn’t have everything promonpromon does does

Poorly documentedPoorly documentedDan Foreman’s VST Guide

Must write the code yourselfMust write the code yourself

Some tables are hazardous to good Some tables are hazardous to good performance: _lockperformance: _lock

BravePoint 2011

10

DB Monitoring - GrowthDB Monitoring - Growth

DB ExtendsDB ExtendsGoal: zero – i.e. all growth occurs in fixed size extents

Not as important as it used to be IF there is a very high quality disk storage system

BravePoint 2011

11

DB Monitoring – I/ODB Monitoring – I/O

DB Requests - # of block accessesDB Requests - # of block accesses

DB Reads - # of reads from DB on DB Reads - # of reads from DB on disk required to satisfy DB Requestsdisk required to satisfy DB Requests

DB LevelDB Level

Individual Client LevelIndividual Client Level

Continued…Continued…

BravePoint 2011

12

DB Monitoring – I/ODB Monitoring – I/O

Hit Ratio – Ratio of DB Requests:DB Hit Ratio – Ratio of DB Requests:DB ReadsReads

Goal: find as many blocks in Buffer Cache (-B) as possible; i.e. High Hit Ratio

Fix: Dump&Load

Fix: Increase Buffer Cache

Fix: Avoid ‘stupid’ queries (no good index, ODBC reporting, etc.)

BravePoint 2011

13

DB Monitoring – Before ImageDB Monitoring – Before Image

Buffers FlushedBuffers FlushedGoal: Zero (or near zero)

Fix: BI Cluster Size

Fix: Async Page Writers (APWs)

Empty Buffer WaitsEmpty Buffer WaitsGoal: Zero

Fix: BI Buffers and/or BI Writer (BIW)

BravePoint 2011

14

DB Monitoring – Before ImageDB Monitoring – Before Image

Long Duration TransactionsLong Duration TransactionsGoal: 60 minutes maximum

Fix: good code

BI SizeBI SizeGoal: “reasonable” – typically 1-4gb max

Fix: good code

BravePoint 2011

15

DB Monitoring – After ImageDB Monitoring – After Image

Empty Buffer WaitsEmpty Buffer WaitsNot running After Imaging – don’t be a fool

Goal: Zero

Fix: AI Buffers and/or AI Writer (AIW)

BravePoint 2011

16

DB Monitoring – Record LockingDB Monitoring – Record Locking

Lock Table High Water Mark (HWM)Lock Table High Water Mark (HWM)Goal: don’t hit the HWM

Fix: larger Lock Table (-L) – be careful!!!

Fix: fix the code

Record WaitsRecord WaitsGoal: Zero

Fix: fix the code

BravePoint 2011

17

DB Monitoring - ContentionDB Monitoring - Contention

Latch Contention – Latch TimeoutsLatch Contention – Latch TimeoutsGoal: Low (zero is impossible)

Fix: adjust –spin parameter

Semaphore Contention – Semaphore Semaphore Contention – Semaphore WaitsWaits

Goal: Low (zero is impossible)

Fix: adjust –semsets parameter

BravePoint 2011

18

DB Monitoring - HealthDB Monitoring - Health

Log (.lg) FileLog (.lg) FileMonitoring for Warnings & Errors

Blocks & ChainsBlocks & Chainsproutil <db> -C dbanalys

Avoid running during production hours

Run against a copy of production DB

Several other health checking options exist

Continued…Continued…

BravePoint 2011

19

DB Monitoring - HealthDB Monitoring - Health

Pointers (Indexes Pointers (Indexes Data) Data)proutil idxcheck

proutil idxfix

BravePoint 2011

20

DB Monitoring - StructureDB Monitoring - Structure

Tables or Indexes in the Schema AreaTables or Indexes in the Schema Area

Indexes in Table AreasIndexes in Table Areas

Tables in Index AreasTables in Index Areas

Type 2 Areas!!!Type 2 Areas!!!Sorry but some QAD staff are just plain wrong about T2 Areas

BravePoint 2011

21

DB Monitoring - DemographicsDB Monitoring - DemographicsScatter FactorScatter Factor

proutil dbanalys/tabanalys

4+ for large, heavily used tables is EVIL

Bug in early V10 Scatter Factors

Continued…Continued…

BravePoint 2011

22

DB Monitoring - DemographicsDB Monitoring - Demographics

FragmentationFragmentationproutil dbanalys/tabanalys

Fragmented Records require a minimum of 2X more I/O to read and write

Continued…Continued…

BravePoint 2011

23

DB Monitoring - DemographicsDB Monitoring - Demographics

Tables getting large enough to Tables getting large enough to warrant migration to a dedicated warrant migration to a dedicated Storage AreaStorage Area

proutil dbanalys/tabanalys

Index Utilization PercentIndex Utilization Percentproutil dbanalys/ixanalys

BravePoint 2011

24

Progress Client MonitoringProgress Client Monitoring

I/OI/O_tablestat & _indexstat VSTs

_usertablestat & _userindexstat VSTs

Code EfficiencyCode EfficiencyProgress Application Profiler – undocmented

What Program is being executedWhat Program is being executed_connect VST (since V10.1C)

BravePoint 2011

25

SQL Client MonitoringSQL Client Monitoring

I/O – same as Progress ClientsI/O – same as Progress Clients

Query PlanQuery Plan

UPDATE STATISTICS – not really UPDATE STATISTICS – not really monitoring but tuningmonitoring but tuning

BravePoint 2011

26

Monitoring - OtherMonitoring - Other

Memory consumptionMemory consumptionProgress Clients

Java

AppServers & WebSpeed

TomcatTomcatDoes it respond?

BravePoint 2011

27

ConclusionConclusion

Questions? Questions?

If you need further assistance:If you need further assistance:Dan Foreman Publications

ProMonitor

www.BravePoint.com

danf@prodb.comdanf@prodb.com

Thank You For Coming! Don’t forget Thank You For Coming! Don’t forget your Conference Evaluations!your Conference Evaluations!

BravePoint 2011

28