costpoint with oracle database - microsoft · 2019-11-21 · » oracle tuning and diagnostics pack...

CP-63Costpoint with Oracle Database

Steve Sisak, Scott Wilburn Deltek

Oracle Topics

SI 566450 - CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 2

» Introduction

» Oracle Components for Costpoint

» Performance Settings

» Memory

» CPU

» IOPS – Input Output Operations per

Second

» Virtual Environments

» Degree of Parallelism and Partitioning

» Technical Troubleshooting

» W_App_Status_Info table

» Tracing

» The Application Workload Repository

Report (AWR)

Introduction

CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 3

» Technical session on Oracle database use and configuration.

» Presented from the perspective that these are some of the most common causes of performance

issues.

» Will hopefully help you understand all the other factors that can be part of performance issues.

» Best suited to:

» Application administrators interfacing with DBA’s.

» DBA staff responsible for infrastructure related to Costpoint.

» Accountants who would like to understand more about this topic.

» Separate sessions available for SQL Server and Advanced System Management (Weblogic).

Oracle Components for Costpoint


» Costpoint normally tests and supports two late releases of the Oracle database. With multiple supported versions of Costpoint this can result in 3 versions of Oracle supported in some form. Costpoint has several performance optimizations that benefit from specific Oracle features. Deltek recommends considering these features for optimal performance.

» Enterprise Edition – Gives parallel execution support.

» Partitioning Option – Supported to optimize some of the critical month end Costpoint processes.

» Oracle Tuning and Diagnostics Pack – Needed for Application Workload Repository reports. These are the number one performance management tool used by Customer Support and Engineering.

» Nice to have features include:

» Real Application Clusters (RAC).

» Features in use but not regularly tested by Deltek.

» Data Guard.

» Multitenant.

» Oracle database appliances / Exadata.

Performance Settings - Memory


» Memory and Caching are key to both OS and Database performance.

» Caching is the primary way to avoid IO bottlenecks.

» Set minimum and maximum database memory to be the same value.

» Set minimum values close to expected requirements for key memory pools like SGA.

» Always leave 25% of the OS memory free and not committed to the database.

» Corporate applications running on the server might require adding memory.

» Add memory if either the database or OS show signs of memory paging to disk.

» The Advisory Statistics section of the AWR has the most important memory information.

Performance Settings - CPU


» Always use minimum of 2 Ghz clock speed for CPU

» Avoid hyper threading when counting your CPU cores it impacts DOP.

» For most mid sized databases 2 to 4 CPU’s is the minimum today. With modern CPU’s a 2 CPU server can have 12 to 20 physical CPU cores.

» OS issues like the Meltdown and Spectre security patches took away 20 – 30% of CPU performance.

» Size beyond the database since the OS and other system applications need CPU to operate after all database needs are met. As with memory have 25% extra capacity.

» IO performance is the first impact if the database uses all the CPU resources.

» Need to monitor peak workload time periods.

» Make use of antivirus exclusions or switch from real time to scheduled scans.

» Never use Power Saving settings in the system BIOS, a Virtual Machine guest OS, or a Host OS.

» Deltek minimum CPU recommendations are based on physical servers and virtual CPU’s can be much different than physical ones.

IOPS – Input/Output Operations per Second

CONFIDENTIAL © Deltek, Inc. All Rights Reserved.7

» Standard measurement of storage system performance. One of the most common bottlenecks.

» Faster / “More” IOPS = Faster Costpoint or more concurrent users.

» Direct 1 to 1 correlation between IOPS increases and performance of key Costpoint closing processes in Deltek’s performance lab.

» Impacts of Memory and CPU tuning on a database are minimal until IO is adequate.

» Sub 10 ms latency is target for database IO.

» Deltek performance lab database servers have current IOPS ratings of :

» 17,000 – decommissioned in 2018 but was the standard for 7 years.

» 30,000 – 40,000 internal low performance SSD array.

» 70,000 – internal high performance SSD array.

» 106,000 – Nimble AF60 all flash storage array, dual 10Gb active Ethernet storage network. Highest performance.

» Deltek uses Diskspd utility to measure IOPS. (Microsoft Technet). Available through Customer Care.

» Deltek also has a database driven load test that is not Windows specific that can be used to test IO.

IOPS and Hosting


» IOPS are the most important factor for a database system like Costpoint.

» IOPS are critical for performance but often overlooked selecting instance types.

» Hosted IOPS are complex due to options to dedicate or “burst” and different limits that are tied to IOPS like throughput.

» IOPS is the first sizing parameter to consider. Throughput needs to match / sustain IOPS for the duration of a long running Costpoint process.

» Sustainable IOPS are often lower than Peak IOPS or have other limits like duration. Test and understand any hosting environment used for a database.

» Example:

» C5n.2xlarge AWS instance – 20,000 IOPS (30 min limit). 3,500 Mbps throughput (30 min limit).

» Sustainable IOPS = 10,000. Sustainable throughput = 1,750 Mbps.

» Need to understand that a “Fast” Costpoint system can be a 900 lb gorilla of IO.

Virtual Environments


» Best practices with Virtualization.

» Monitor and Manage to meet the database peak workload requirements.

» There are more variables which make sizing and management more complex.

» Reserve the resources in the VM required for the database server.

» Don’t recommend sharing or pooling CPU or Memory.

» Mandatory to test virtual environments with production workloads.

» Technologies that move virtual machines or their storage can cause problems for the

Weblogic cluster and database. The short system disconnects or suspension they cause

have taken systems offline.

Applications Tuned for Parallelism and Partitions


» These applications were tuned for parallel

execution and partitioning and are the only jobs

tested in a parallel job queue.

» Parallelism and Partitioning work together and

have been supported and tuned together with

Oracle since Costpoint 6.1.

» Very significant performance improvements

were realized by this tuning.

Degree of Parallelism


» A calculation by the database based on CPU and database configuration settings.

» Used by Costpoint to optimize performance in conjunction with Partitioning.

» Controlled through the Job queue. Job queues default to serial but have a parallel attribute that can be set. Parallelism isnot used unless parallel job queues are configured.

» An undocumented parameter is available, for Oracle databases only, that allows the DOP to be set for parallel job queues through Weblogic.

» This configurability is not supported on SQL Server.

» On the Weblogic server you add the following setting to the enterprise.properties file, with a non zero value, to set the DOP of any parallel queue running on that server. Requires Sysjar 46 or later.

» “System Name”.parallelSqlLimit=8 - Sets Dop to 8 for jobs running through a parallel queue.

» Future KB on the topic KB 99594

» This allows easy management and testing of the DOP that can be supported by the CPU and IO of a Costpoint Oracle database system.

» Don’t use any other methods to attempt to control parallelism. Don’t assign parallel attribute to tables or indexes.

» In Oracle RAC it is mandatory that all parallel jobs be pinned to the same server. They will not perform well if distributed.

» Parallel_force_local = True

Partitioned Tables


» The following tables can be partitioned with an optional partitioning script, supplied with the

Oracle installation, which is tested exclusively for Oracle performance.

» Most are partitioned by PD, and possibly Sub_Pd. A FY partition can be safely added but

was not a part of the partition scripts. FY and any other partition are the customer’s

responsibility to test and maintain.

Technical Troubleshooting


» Information about the issue.

» Complete description.

» AWR.

» Unusual Wait Events?

» Is the Costpoint SQL the top SQL?

» Application Trace / Tkprof output.

» Is it repeatable and how to repeat?

» Has anything changed?

» A system diagram can summarize the architecture and improve troubleshooting.

» Weblogic diagnostic image.

» Check for errors that happen at the time of the incident.

» W_App_Status report for the impacted process will give the performance history.

W_App_Status_Info table


» Automatically stores run time information for all reports and processes run in Costpoint

» The data can be accessed in any screen with an Actions / Report button on the menu.

» A Costpoint report interface allows you to quickly run reports for any Costpoint job like Calculate Revenue. Reports can also be run by time period and in that case you get a list of everything in Costpoint that ran at a particular time.

» Great way to track run time changes in key jobs.

» Answers the question “what else was running” also records other key data like application server used and how the job was submitted.

» Should be used to double check time periods being used for AWR reports to make sure the correct job was running.

» Supply both baselines and trends for Costpoint processes that can help evaluate tuning or hardware impacts.

W_App_Status_Info data


» APP_ID, ACTION_ID and RPT_ID – Identify the work.

» COMPLETE_FL – Completion status.

» EXEC_METHOD_CD – How job was submitted ( UI, API, Async, PrcSrv).

» USER_ID – Submitter.

» SYS_TEXT – Job messages usually completion or error.

» JOB_START_DT, JOB_END_DT or ACTION_START_DT, ACTION_END_DT – Start and

stop times and dates to allow calculation and tracking of elapsed time.

» COMPANY_ID – Company run for.

» SERVER_ID – Application server doing the work.

Weblogic Trace Options


» traceSlowStatements.

» Maintenance screen only trace for special situations.

» enableSQLTrace.

» Weblogic based application text trace. No DBA involvement but only elapsed time from each statement. No database impact.

» enableOracleTrace.

» Oracle specific trace details – DBA required uses Oracle Trace / Tkprof facilities. Traces the specific application by starting the database trace from Weblogic. Will trace every time the application is run. Critical for performance analysis.

» Use Trcsess utility to combine multiple trace files from parallel applications before Tkprof. (No KB yet).

» For full details see KB’s 96692, 85245, 75574.

» Most universal for database performance is enableSQLTrace.

» Traces are for 1 application per entry in the enterprise.properties file.

» In special situations for maintenance screens use traceSlowStatements.

enableSQLTrace Output


» Comparison of the same statement in two time periods.» 71. Statement:

» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,

SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*

C:1 R:All */

» First Executed on: Sat Jun 01 07:06:35 CDT 2019

» Parsed: 8

» Executed : 8

» Has literals: YES

» Total Exec Time: 12485139 milliseconds (200 minutes approximately)

» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)

» 71. Statement:

» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,

SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*

C:1 R:All */

» First Executed on: Tue Jun 04 07:06:48 CDT 2019

» Parsed: 8

» Executed : 8

» Has literals: YES

» Total Exec Time: 226934 milliseconds (4 minutes approximately)

» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)

enableOracleTrace Output


» Trace output must be run through Tkprof utility before sending to Deltek.

AWR - Overview data


» Always confirm the date and time. Remember W_App_Status can help. Make sure the report includes the time period that an issue occurred.

» Most reports are run for one or two hours at most.

» Make sure to have a long enough retention period to allow going back to get this data.

» Keep snapshots of all month end and peak periods for reference (Baselines).

» By reviewing these reports you will know what’s normal for your system.

AWR - Top Wait Events


» Wait events are normal and consistent. This section helps spot unusual or abnormal delays in the system.

» New top wait events should always be investigated especially after upgrades or if performance issues are occurring.

AWR - SQL ordered by Elapsed Time


» Key for identifying problem SQL and making sure there are not other SQL interferences.

» Costpoint applications being reviewed should be the longest running SQL.

AWR - SQL ordered by IO Wait Time


» Helps identify problem SQL and determining if IO is part of the issue.

» Used with Tablespace IO statistics.

AWR - Log Switches


» Good for this metric is less than 6 per hour. It usually indicates the log files are too small.

» There are a couple of other important metrics like:

» Sorts (disk) – Instance Activity – Ideally 0.

» PGA Cache Hit % - Ideally mid to high 90’s

» From your baseline look for anything abnormal. The AWR includes a report comparing a current period to a baseline automatically.

AWR - Tablespace IO


» IO performance on reads with less than 10ms is crucial for performance

» If you don’t have this section in Oracle 12.2 you need to update your statistics settings. Oracle KB article available.

AWR - Memory Pool Data


» Buffer Pool, Shared Pool and PGA Advisories are very important to gauge database

memory.

AWR - Oracle initialization parameters that are not default values.


» Undocumented parameters are normally directions from Oracle to address issues and may be asked about.


AWR - Baselines» In coordination with accounting and using the W_App_Status report, run an AWR that covers

the month end processing for each period of the year.

» Save January and December AWR’s as baselines as a minimum.

» Use them to review performance during the year

» Evaluate intermittent performance issues.

» Do capacity planning.

» Evaluate performance tests and configuration or hardware changes.

» For your longest running process supplement the AWR with a trace. Having a known picture

of good performance is extremely valuable in troubleshooting.


Q & A

costpoint with oracle database - microsoft · 2019-11-21 · » oracle tuning and diagnostics pack...

Documents