2000-01-28 strange - perf. for peoplesoft apps

Performance Enhancement Techniques for

Peoplesoft Applications on Oracle DatabasesVersion 3.2 February 7 2000

Paper # 507 – IOUG-A Live 2000

Authors: James E. Strange & Raman R. Batra

Abstract:

The paper covers a range of methods for improving the performance of Peoplesoft HR applications on an Oracle database. It includes information on implementing the cost-based optimizer along with a variety of indexing strategies and other information. The paper presents innovative ways to enhance the performance of Peoplesoft HR applications, encompassing Batch, OLTP and DSS environments. The authors also share documented performance enhancements and benchmark information.

The paper is a team effort of two Oracle Database Administrators who met on-line soon after the formation of the Peoplesoft-SIG of the IOUG-A. After the group started, they began to exchange notes and information regarding Peoplesoft HR on Oracle. The database environments at both sites are somewhat similar - in terms of the size of the database and organizational size, however the workload and on each database environment is as different as the business needs which drive the use of the application. The similarity of the techniques used in both environments, mainly implementation of the Cost-Based Optimizer, was striking. The authors decided to share their experiences with the Oracle community in the form of this publication.

Table of Contents

1 Introduction:..............................................................................................................................4

2 General Tuning and Architectural Considerations:...................................................................4

3 Identifying SQL and Performance Bottlenecks:........................................................................4

4 The Oracle Optimizer................................................................................................................5

4.1 Driving Tables, Join Orders and Optimizer Modes:..........................................................6

4.2 Choosing an Oracle Optimizer:........................................................................................10

4.2.1 Setting the Optimizer Mode:.....................................................................................11

4.2.2 Collecting Statistics:.................................................................................................12

4.2.3 Ongoing Maintenance with the Cost-Based Optimizer:...........................................13

4.2.4 Preparation Before Tuning:.......................................................................................13

5 OnLine Access.........................................................................................................................13

5.1 General Information.........................................................................................................13

5.2 Case Study........................................................................................................................14

6 Batch Jobs................................................................................................................................14


6.2 Payroll Calc Case Study...................................................................................................15

6.2.1 Actions taken............................................................................................................15

6.3 Final Results.....................................................................................................................30

7 Decision Support System........................................................................................................30


7.2 The Infamous Security Tree.............................................................................................31

7.3 Case Studies.....................................................................................................................31

7.4 Final Results.....................................................................................................................44

8 Conclusions and Summary......................................................................................................44

9 Appendix A: Scripts................................................................................................................45

10 Appendix B: New Indexes...................................................................................................48

11 Appendix C: Test Environments..........................................................................................50

12 Appendix D: Recommended References.............................................................................51

2

Table of Figures

Figure 1: A simple query joining three-tables and the RBO execution plan....................................7

Figure 2: Query with table order switched in the FROM clause for RBO.......................................8

Figure 3: Query with cost-based optimizer.......................................................................................9

Figure 4: Query with table order switched in the FROM clause for CBO.......................................9

Figure 5: Panel Access Speeds.......................................................................................................14

Figure 6: Calc Query.......................................................................................................................15

Figure 7: Calc Tables......................................................................................................................16

Figure 8: Initial Explain..................................................................................................................17

Figure 9: PS_JOB1_NDX...............................................................................................................19

Figure 10: Bit Map Explain............................................................................................................19

Figure 11: TKPROF........................................................................................................................22

Figure 12: PS_PAY_LINE1_NDX.................................................................................................27

Figure 13: Final Tkprof for CALC.................................................................................................27

Figure 14: Run-time comparison of 12 queries between Rule-Based Optimizer and Cost-Based Optimizer.................................................................................................................................32

Figure 15:PS_PERSONAL_SRCH View.......................................................................................32

Figure 16: Details of Query2 (without hinted PS_PERSONAL_SRCH_Q View)........................32

Figure 17: Details of Query2 (with hinted PS_PERSONAL_SRCH_Q view)..............................39

3

1 Introduction:

Access to Peoplesoft HR Applications (HR, Benefits, Payroll, Self-Service etc.) on an Oracle Database can be classified into three major categories: Batch Jobs (Payrolls, SQR's/Reports), Online Access (OLTP - Panel Users) and Decision Support (PSQUERY and custom reports.) The following sections present tuning studies and suggestions for each of the categories including test results, case studies and associated data. Information about Batch Jobs and Online Access was provided by James Strange. Information about DSS systems was provided by Raman Batra.

2 General Tuning and Architectural Considerations:

It must be assumed that the database server and instance are tuned properly. Before exploring relatively advanced options such as implementing the Cost Based Optimizer, the SGA must be sized so that the critical hit ratios - database buffer hit ratios, library cache hit ratio, data dictionary cache hit ratios, are optimized. Peoplesoft can be CPU intensive, so size your system accordingly. In addition to the SGA size, you may also need to look at the total memory on the database server and your concurrent user population. Other issues such as the number of dedicated two-tier connections and logical/physical three-tiers also may affect the CPU and memory available on the server. Each dedicated connection (two-tier) takes from 6 to 10 MB of memory on the server, depending upon the platform of Oracle and the sort_area size/ hash_area_size, and memory used per connection. It would be a good idea not to give ALTER SESSION privilege to the general user community, since there is the potential for misuse. A big sort can take a large chunk of the PGA (private global area) and eventually consume all available memory.

In addition to CPU and memory, you may also need to ensure your disk system uses hardware or software mirroring (or both) and the IO is balanced. If possible, use the Oracle recommended OFA architecture as much as possible to avoid contention between data, index, rollback, and temporary segments and redo logs. Sizing of rollback segments can be crucial depending upon the size of the batch jobs and their frequency. Also, last but not the least, do not forget sizing of temporary tablespaces, since some temporary activity will always need disk and may not fit into shared memory. Be sure to declare the temporary tablespaces as TEMPORARY in order to improve space management for sorts.

3 Identifying SQL and Performance Bottlenecks:

You may need to set up procedures and/or scripts or use tools to identify and isolate bottlenecks. These could be as simple as querying the server OS environment for the top 10 resource intensive processes (CPU and memory) and/or querying V$SQLAREA with limits set for buffer gets etc. … You could also use third-party tools or the tuning tools available with the OEM Performance Pack. At Raman’s employer, the HR/IT team set up a CGI script to create a web-page showing CPU and memory utilization and resource availability.

Jim uses several different ways to monitor the databases at his employer. These include OEM, SQLab, and a number of home grown SQL scripts. In order to get a quick snapshot of the users currently connected to the database, one can use the script allActive.sql which is included in the appendix. The script sql.sql extracts the current SQL for a specified sid from the V$SQLTEXT

4

table. Finally, the scripts makeExplain.sql and exp.sql can be used to obtain the execution plans for the SQL associated with a particular sid.

No matter what method used to monitor your system, you must be able to identify problem areas, especially during payroll batch runs. Thomas (1988)1 has published an excellent roadmap for identifying bottlenecks and tuning your Oracle server and database environments for Peoplesoft Applications. In particular the techniques used to identify and isolate performance bottlenecks are definitely worth reading. Peoplesoft Corporation seems to have increased the number of bind variables with release 7.x of Peoplesoft as compared to release 6.x, so some time is saved parsing SQL statements with different variable values, which helps performance somewhat. In addition, the bind variables help reduce contention in the buffer cache.

4 The Oracle Optimizer

After a SQL statement is parsed for syntax accuracy and object dependencies, it is processed by the Oracle Optimizer to generate an optimal path or execution sequence. It is at this point that the Optimizer Mode can greatly affect the access paths, processing, and ultimately the performance and resource utilization needed by a SQL statement. The Oracle Database basically has two optimizers:

1. Cost Based Optimizer (CBO): The Cost-Based Optimizer was introduced by Oracle about the time of release 7.2. This feature came in response to similar moves by other database vendors. The CBO received a large amount of adverse reaction from the user community. Much of that reaction was well deserved. The early versions of the cost based optimizer were notoriously unpredictable. It frequently chose execution paths inferior to the rule-based optimizer. However, each successive version of the database server has included enhancements to the cost based optimizer. Gradually, the cost based optimizer has improved. It can now be a vital element to improving the performance or your system.

The CBO calculates the relative cost of a query by trying several combinations of query paths, selecting the one with the lowest cost. Still, it is not perfect and sometimes there can be no substitute for human intelligence, especially if knowledge of business processes can help you take decisions on schema design. However, the CBO is generally effective at choosing the best access path when implemented and supported correctly.

Another important consideration is that the CBO is a requirement for using the new advanced features of Oracle. All the new features of Oracle 8.i (materialized views, function indexes, hash-partitioning), and some features of Oracle 8, such as range-partitioning and parallel options, work only with the CBO. In order to take advantage of the latest and greatest features in scalability and performance, you may need to implement the CBO.

2. Rule Based Optimizer (RBO): The Rule-Based Optimizer is the "old faithful" way of Oracle processing, and a very predictable way of choosing a query plan. It works exactly the same way even if a table has 10 rows or 10 million rows, which can be a blessing or a curse, depending upon several factors, most importantly your data distribution. The RBO is not aware of the distribution of values in a data-segment and does not take advantage of histograms, which track the dispersion of values in a data and/or index segment. The RBO uses a predefined set of rules

5

based on precedence logic to figure out what access path to use. It is driven by about 15 precedence rankings (with single row access by rowid at the very top and full table scan right at the bottom of the rankings. Several good tuning publications 2,3,4 detail these precedence rankings. The rule-based optimizer chooses an access path based on:

Available access path for the statement

Ranks of the access paths

The WHERE clause of the SQL statements and the order of the table-names in the FROM clause help the RBO to determine possible access paths. The optimizer then chooses the highest ranked access path. Access by index is always ranked higher than tablescans. As a result, the optimizer will always prefer an access path which uses an index. In some cases that approach can be a problem. Sometimes a full table scan may execute faster than access by index. The usual solution for a bad plan with the RBO is to place hints in the query, which will direct the optimizer to a better plan. Unfortunately, that solution is frequently not available, and most ERP systems, including Peoplesoft, do not encourage you to change the code even where possible. In the case of Peoplesoft’s PSQUERY tool, it is effectively impossible to add hints to queries.

If you do not have a separate environment for decision support, you could have a few "queries from hell" consuming most of your system resources and dragging down the online and payroll processes. It is mostly in decision support and OLTP where you will find the biggest limitations of the RBO. As Jim found out with online access, the improvements were dramatic after the RBO was replaced with the CBO.

4.1 Driving Tables, Join Orders and Optimizer Modes:

The driving table is the table processed first in a query. The rows retrieved from the driving table are matched with the other tables in the FROM clause. One of the most annoying limitations of the RBO is the heavy dependence on the order in which tables are joined in the SQL statement. The execution plan can be changed dramatically by reordering the tables in the query. That can have terrible results in an online environment using tools such as PSQUERY where the SQL statement is created by people who have no idea how the database is designed.

From our experience, the Cost Based Optimizer does not seem to be dependent on the order of the tables in the query, as is the RBO. The driving tables for the RBO or CBO can be determined as follows:

In the RBO, the last table in the FROM clause acts as a driving table, all things being equal.

6

In the CBO, according to published literature, the first table in the FROM clause acts as a driving table, unless the optimizer determines a different plan will perform better. This again depends upon how the tables are joined and if the join results in a table scan, unique index scan or index range scan. We found the Cost-Based Optimizer generated the same execution plan, no matter what table order was used in the FROM clause. The following figures illustrate the effect of changing the table order for the RBO.

Figure 1: A simple query joining three-tables and the RBO execution plan

select a.name, a.phone, b.empl_status, b.effdt, c.empl_rcd#, c.eff_status

from

sysadm.ps_personal_data a,

sysadm.ps_direct_deposit c,

sysadm.ps_job b

where a.emplid='99999'

and c.emplid=a.emplid

and a.emplid=b.emplid


The execution plan for this with the rule-based optimizer is:

Query Plan

----------

SELECT STATEMENT Cost =

NESTED LOOPS

NESTED LOOPS

TABLE ACCESS BY INDEX ROWID PS_PERSONAL_DATA

INDEX UNIQUE SCAN PS_PERSONAL_DATA

TABLE ACCESS BY INDEX ROWID PS_JOB

INDEX RANGE SCAN PS#JOB

TABLE ACCESS BY INDEX ROWID PS_DIRECT_DEPOSIT

INDEX RANGE SCAN PS_DIRECT_DEPOSIT

In the above plan the first table processed is PS_PERSONAL_DATA, followed by PS_JOB, the results are put into a nested loop and the result set is matched with data from the third table -

7

PS_DIRECT_DEPOSIT. The driving table, or the table first processed will always be PS_PERSONAL_DATA because of the clause a.emplid='99999'.

Figure 2: Query with table order switched in the FROM clause for RBO


from sysadm.ps_job b,


sysadm.ps_personal_data a





Execution Plan for the above query:

Query Plan

----------


NESTED LOOPS

NESTED LOOPS







In the figure, the first table processed is PS_PERSONAL_DATA, followed by PS_DIRECT_DEPOSIT, the results are put into a nested loop and the result set is matched with data from the third table - PS_JOB. The driving table will always be PS_PERSONAL_DATA because of the clause a.emplid='99999'. The order of tables in the query will affect the execution plan. As a result, although the final result is the same, the performance can vary tremendously. It can make a huge difference; especially if a full table scan is performed on a table with 10 million rows as compared to a table with say 10000 rows of data. The following figures demonstrate how changing the table order does not effect the plan for the CBO.

8

Figure 3: Query with cost-based optimizer


from

sysadm.ps_personal_data a,


sysadm.ps_job b





Query Plan

----------

SELECT STATEMENT Cost = 9

NESTED LOOPS

NESTED LOOPS







Figure 4: Query with table order switched in the FROM clause for CBO


from sysadm.ps_job b,


sysadm.ps_personal_data a





9

Query Plan

----------


NESTED LOOPS

NESTED LOOPS







This behavior can be both a blessing and a curse. When dealing with Decision Support Environments, the person writing the queries is intent on getting the business logic and typically does not want be concerned about the order of tables in a query. In fact, quite a few queries are constantly changed as managers or end-users add requirements. This frequently results in complicated FROM and AND clauses, and ever growing numbers of columns in the SELECT statement. In such a case, the CBO frees the developer from worrying about the table order. This also holds true for adhoc tools such as PSQUERY. Unfortunately, the CBO can hurt performance in some circumstances. If it does not have the right statistics or chooses a bad execution path, performance can be very bad.

In Raman’s system, which is a mixed OLTP, Decision Support and Batch environment, frequently they came across "queries from hell" which would take hours, even days to complete. Simply re-ordering the tables in the join made a tremendous difference under the RBO.

Comparing the two optimizer modes is somewhat like comparing an automatic transmission (CBO) to a manual transmission (RBO). The RBO, maintained by Oracle Corporation for backward compatibility, gives developers a lot of control over the way SQL statements execute. However, in most ERP applications, and Peoplesoft is no exception, it sometimes is impossible to modify SQL, so the optimizer to make intelligent selections to satisfy rather generic queries. That is where the CBO can be of crucial benefit.

4.2 Choosing an Oracle Optimizer:

The Oracle default optimizer is Choose. This simply means the system will run CBO whenever there are statistics available. Otherwise it will use the RBO. If one contacts Peoplesoft, their technical support staff will automatically advise setting the database to RULES mode. Why do

10

they advise using RULES? Peoplesoft is not in the database business. They operate on a number of platforms and have little desire to keep up with changes to Oracle. Plus, as mentioned before, the early versions of the CBO were justifiably scorned.

Starting with Oracle 7.3.4 things have changed. Our tests have indicated that significant performance gains may be achieved by going against the advice of Peoplesoft and using the CBO set to first_rows mode. However, the unfortunate truth is that while the CBO has great potential for improving performance, it can also take a good deal of effort to make it work well for your environment. If your shop has a number of custom reports or jobs tuned to work with the RBO, changing to CBO may in fact hurt the performance of those jobs.

As an example, Jim implemented cost based optimization at his employer. They experienced increased performance for nearly all online access. Panels, which formerly required 20+ seconds would appear in 2 or 3 seconds using the cost, based optimizer. This was greeted by a great deal of enthusiasm by the customers. However, when the batch cycle ran for payroll… the party ended. Payroll was extremely slow. So, they restarted the database in order to switch back to rules. The payroll then finished in a reasonable amount of time.

Eventually Jim was able to identify and cure the problems with the batch jobs. Their shop now runs in CBO 24 hours a day. However, this serves to illustrate that implementing the CBO can be a challenge, initially requiring significant work from the DBA.

4.2.1 Setting the Optimizer Mode:

The Optimizer Mode can be set at the following levels:

1. Database Instance Level - this sets the defaults for all sessions connected to the database. (set in the init[SID].ora by the DBA).

RULE: Sets up default optimizer mode as Rule for every connection to the instance. Set OPTIMIZER_MODE=rule in your init.ora file.

CHOOSE: This is the default. If you do assign any value to the initialization parameter OPTIMIZER_MODE, CHOOSE is the default. If statistics are present, the optimizer will try cost-based optimization. If no statistics are present, it will try rule-based optimization. The situation can worsen, when we have statistics present on some tables and indexes and missing on others. In this case, the cost-based optimizer tries to assume statistics, which can sometimes lead to all kinds of unexplained behavior. It is generally a very bad idea to leave the database in CHOOSE unless statistics are properly maintained.

FIRST_ROWS: These invoke the Cost-Based Optimizer. FIRST_ROWS is more useful to online access situations, where a partial result of the query is presented to the user. This mode is somewhat similar to the rule-based optimizer as far as row operations go, and has a tendency to use more nested loops, which presents data as it is being processed by result sets.

Set OPTIMIZER_MODE=FIRST_ROWS in your init.ora

11

ALL_ROWS: ALL_ROWS is better for batch situations where you want the batch job to complete, but you don't necessarily want to monitor data being processed. You may find that the percentage of hash-joins increases as compared to nested loops.

In our experience, we found FIRST_ROWS to give superior results to ALL_ROWS for Peoplesoft applications.

2. Session Level - Within each session, you can change the optimizer mode, as long as you have the ALTER SESSION privilege. In Peoplesoft, it is very hard use this option with online panels. It is relatively easy to insert an ALTER SESSION in the SQR's. Alternatively you can change the optimizer mode in the initialization file of the SQR (Ask Dustan/Kendall). However, either of these approaches can lead to a lot of customizations in SQRs, which makes it a questionable course of action.When starting to implement the CBO, one approach is to set the default mode for the instance as FIRST_ROWS and use RULE for the batch jobs via the ALTER SESSION command until you have tuned them.

An example would be:

SQL>ALTER SESSION SET OPTIMIZER_GOAL=RULE;

3. Code Level (using Hints) - In a SQL statement, you can give hints. Hints tell the optimizer what access path you want it to use. You can force full-table scans, or force the statement to use a specific index. Unless you use the RULE hint, almost every hint uses cost-based optimization.

The optimum way would be to set the init.ora to a particular mode and then let every process - online, batch and decision support to default to that optimizer mode. However, as you transition to the CBO, you may use some or all of the above methods to change your optimizer settings.

4.2.2 Collecting Statistics:

The CBO needs statistics on every data segment, for Peoplesoft everything owned by the schema owner must be analyzed (in most cases, this will be SYSADM). The objects owned by SYS and SYSTEM should not be analyzed, as recommended by Oracle, especially if the optimizer mode is set to choose.There are several ways to collect statistics:

1. ANALYZE TABLE SYSADM.PSTREENODE ESTIMATE STATISTICS sample 30 percent; (this will estimate statistics with a sample of 30% of table size, you can compute them too)

2. ANALYZE INDEX SYSADM.PSTREENODE COMPUTE STATISTICS; This computes statistics for indexes, to be used when you rebuild indexes or alter/create new indexes.

12

3. Execute DBMS_UTILITY.ANALYZE_SCHEMA('SYSADM','COMPUTE',NULL,30);This Oracle supplied package will analyze the entire schema, tables and indexes both, and you can specify the method (compute or estimate).

Depending upon the volatility of the data, you may increase the sample to 100 percent or to a specified number of rows, in most cases 25-30% sample is enough for the optimizer to work accurately. Computing statistics is CPU and memory intensive and will take a fair amount of temp space, so it is recommended this be performed during off-hours. You will need statistics for tables and indexes. Collecting statistics for tables will also collect index statistics, however if you rebuild an index or create a new one, you will need to collect statistics for the rebuilt/new/altered indexes.

4.2.3 Ongoing Maintenance with the Cost-Based Optimizer:

Once you start to use the cost-based optimizer, you will have to ensure statistics are present and fairly current (once a week should be enough for most medium sized systems). With Peoplesoft you may be performing Peopletools upgrades and/or tax upgrades several times a year. These are the occasions you when you really need to be careful. All it takes is one table missing statistics, to totally confound and upset your query run-times.

4.2.4 Preparation Before Tuning:

It is highly recommended you experiment and read up on tuning indexes and complex joins, since Peoplesoft HR will give you ample opportunity to test your knowledge in those specific areas. In most cases, you cannot change the SQL code, or at least should keep the code "vanilla"; your tuning strategy must be designing and tuning appropriate indexes. Also, experiment with different optimizer modes. Practice in tuning joins and designing indexes with Rule Based and the Cost Based Optimizer would really help. We recommend taking a few simple queries (4-5 tables) and analyze and compare the execution plan for the RBO and the CBO.

The following sections demonstrate performance enhancement techniques for Peoplesoft databases. This includes a look at enhancing online performance by using the cost based optimizer, tuning batch jobs for optimum performance, and tuning DSS queries for Peoplesoft.

5 OnLine Access

5.1 General Information

When the cost based optimizer was implemented at Jim’s employer, there was a significant improvement in online response time. Screens, which formerly required 20 seconds to appear, would appear in less than 10 seconds using cost based. The customer service representatives greeted this with a great deal of enthusiasm. However, when the batch cycle ran for payroll… the party ended. Payroll was extremely slow. So, the database was restarted under rules in order to finish payroll in a reasonable amount of time.

Later Jim, was very surprised to find the users insisting he switch the system back to ‘”the way you had it before payroll.” The user’s perception of online performance under CBO was so much better that they agreed to limit batch jobs to designated windows of time. The database was

13

manually rebooted at the start and end of each batch window in order to run RULES during batch jobs and cost for online access.

Of course, that was not an ideal situation. So, significant effort was devoted to figuring out how to improve batch performance so the system could remain in cost based mode 100% of the time. Eventually, a solution was found. The key is to use FIRST_ROWS along with carefully tuning the indexes in the database. The following case study shows how one job was improved by the use of the CBO.

5.2 Case Study

Testing online performance was a simple matter of timing how long it takes to bring up panels of information from Peoplesoft. In virtually every case tested, the performance of the cost based database was significantly superior to the response time from RBO. It must be noted that in order to be sure caching on the client did not influence the timings, the cache files on the PC were deleted between tests. The following table lists some of the panels accessed and the times observed for each database type.

Figure 5: Panel Access Speeds

Panel Rules Based Cost Based

employee contact info 20 seconds 9 seconds

business expenses 4 seconds 2 seconds

parking expenses 3 seconds 1 second

current job- benefits partic 1st 5 seconds 1 second

benefits 33 seconds 25 seconds

paycheck review 15 seconds 11 seconds

As the data demonstrates, online access is often faster under cost based than under rules. The difference in times is not always dramatic; however, in terms of user perception, the difference is significant.

6 Batch Jobs


As mentioned previously, the cost based optimizer can have a terrible affect on batch jobs for Peoplesoft. However, that can be fixed. The key is to use FIRST_ROWS along with carefully tuning the indexes in the database. The resulting system will produce performance equal or superior to a rule based system. After tuning, the performance will almost always be better than a vanilla Peoplesoft installation using rules.

There are a wide variety of jobs which take place during the payroll cycle. Two of the major jobs will be covered in this section. Those are the payroll jobs: calc and confirm. Calc jobs can be

14

extremely time consuming. To help speed the process, Peoplesoft advises to limit the number of employees in individual paygroups to a very small number. That solution, while workable, is not always very useful. In addition, while it does speed up the process overall, the Calc process itself remains quite inefficient. As it turns out, the SQL used for the Calc jobs is quite similar to the SQL used for the confirm jobs. So, tuning for one is quite likely to improve the performance of the other.

A third type of job is actually a general group of jobs. These are custom reports. Jim’s system has a large number of reports run via SQR against the database. These must be individually tuned to ensure proper performance.

So, Jim set out to capture the SQL running against the database for each of these scenarios. During this effort he used two tools. One was Quest Software’s SQLab Tuner. This allowed you to create a repository, which collects all of the SQL issued during a batch job. The second tool was a series of scripts which capture the SQL for processes running against the database. The scripts also can be used to produce explain plans and tkprof output (see appendix for scripts.)

Although not all of the many enhancements made to the database will be covered by this paper, the following section provides a case study showing how to improve the performance of the calc routines dramatically. The results of a properly tuned cost based database are almost always faster than an RBO database.

6.2 Payroll Calc Case Study

Calc on large paygroups takes an exceptionally long time. As the company increased in size, a step that formerly took 3 to 5 minutes for an average sized company and paygroup began to require ½ hour or more for a single large paygroup. Contacting Peoplesoft did not produce any helpful information. They advised breaking the paygroup into multiple smaller groups. So, it was left to Jim to improve the performance.

6.2.1 Actions taken

Capturing the SQL to identify problem queries was relatively easy using SQLab and a series of homegrown scripts. SQLab was used to create a repository, which collected all SQL executed during the calc job. By simply watching the log output and periodically using the scripts allActive.sql and sql.sql, it became readily apparent that the longest part of the batch job was a single step. Improving the speed of that step became the focus of the tuning effort. For purposes of this paper, the offending SQL statement will be referred to as the CALC statement.

The following figures show the CALC statement, list information about the underlying tables, and show an Explain plan for the CALC statement.

Figure 6: Calc Query

SELECT A.EMPLID

,A.EMPL_RCD#

FROM PS_JOB A

15

,PS_PERSONAL_DATA P

WHERE A.COMPANY='046'

AND A.PAYGROUP='A01'

AND A.EFFDT<='03-DEC-1999'

AND A.EMPL_STATUS IN ('A','P','Q','U')

AND P.EMPLID=A.EMPLID

AND P.PER_STATUS='E'

AND NOT EXISTS (SELECT 'X' FROM PS_JOB F WHERE F.EMPLID=A.EMPLID

AND F.EMPL_RCD#=A.EMPL_RCD#

AND F.EFFDT<='03-DEC-1999'

AND F.EFFDT>A.EFFDT

AND (F.EFFDT<='03-DEC-1999' OR (F.COMPANY='046'

AND F.PAYGROUP='A01'

AND F.EMPL_STATUS IN ('A','P','Q','U')

AND F.EFFSEQ= (SELECT MAX(G.EFFSEQ) FROM PS_JOB G WHERE G.EMPLID=F.EMPLID

AND G.EMPL_RCD#=F.EMPL_RCD#

AND G.EFFDT=F.EFFDT))))

AND A.EFFSEQ= (SELECT MAX(H.EFFSEQ) FROM PS_JOB H WHERE H.EMPLID=A.EMPLID

AND H.EMPL_RCD#=A.EMPL_RCD#

AND H.EFFDT=A.EFFDT)

AND NOT EXISTS (SELECT 'X' FROM PS_PAY_LINE L WHERE L.EMPLID=A.EMPLID

AND L.EMPL_RCD#=A.EMPL_RCD#

AND L.COMPANY='046'

AND L.PAYGROUP='A01'

AND L.PAY_END_DT='03-DEC-1999'

AND L.OFF_CYCLE='N')

ORDER BY A.EMPLID ASC;

Figure 7: Calc Tables

SQL> select num_rows from dba_tables where table_name = 'PS_JOB';

16

NUM_ROWS

----------

233417

SQL> select num_rows from dba_tables where table_name = 'PS_PAY_LINE';

NUM_ROWS

----------

828720

SQL> select num_rows from dba_tables where table_name = 'PS_PERSONAL_DATA';

NUM_ROWS

----------

32057

Figure 8: Initial Explain

SQL> @explain

Query Plan

------------------------------------------------------------------------


SORT ORDER BY

CONCATENATION

FILTER

NESTED LOOPS

TABLE ACCESS FULL PS_JOB

TABLE ACCESS BY ROWID PS_PERSONAL_DATA


TABLE ACCESS BY ROWID PS_PAY_LINE

INDEX RANGE SCAN PS#PAY_LINE

SORT AGGREGATE

INDEX RANGE SCAN PSAJOB

17

FILTER

TABLE ACCESS BY ROWID PS_JOB


SORT AGGREGATE


FILTER

NESTED LOOPS






SORT AGGREGATE


FILTER



SORT AGGREGATE


FILTER

NESTED LOOPS






SORT AGGREGATE


FILTER



18

SORT AGGREGATE


FILTER

NESTED LOOPS






SORT AGGREGATE


FILTER



SORT AGGREGATE


59 rows selected.

The query plan indicated a very costly query, which included a full table scan against PS_JOB. That table scan was obviously a problem, so an index was added to PS_JOB to eliminate the table scan. Both a normal index and a bitmapped index were compared. This situation was an ideal candidate for a bitmapped index. Comparing performance between a normal index and the bit-mapped index, the bit-mapped index produced marginally better performance.

Figure 9: PS_JOB1_NDX

create bitmap index ps_job1_ndx on ps_job(company, paygroup);

analyze index ps_job1_ndx estimate statistics sample 30 percent;

The CALC query performed considerably better with an index. The runtime decreased to 16 to 17 minutes; however, that still was not fast enough.

Figure 10: Bit Map Explain

SQL> @explain

19

Query Plan

------------------------------------------------------------------------


SORT ORDER BY

CONCATENATION

FILTER

NESTED LOOPS


INDEX RANGE SCAN PS_JOB1_NDX





SORT AGGREGATE


FILTER



SORT AGGREGATE


FILTER

NESTED LOOPS







SORT AGGREGATE


FILTER

20



SORT AGGREGATE


FILTER

NESTED LOOPS







SORT AGGREGATE


FILTER



SORT AGGREGATE


FILTER



SORT AGGREGATE


FILTER



SORT AGGREGATE


NESTED LOOPS


21




At this point a comparison to rules mode was made. It turned out that the bit mapped index brought performance up to par with rules based optimizer. The Calc completed in 15:09.08 minutes under rules.

Since it was going to be necessary to better the performance even further, tkprof was used to get more detailed information about the query.

Figure 11: TKPROF

TKPROF: Release 7.3.4.0.0 - Production on Thu Dec 2 10:22:23 1999

Copyright (c) Oracle Corporation 1979, 1996. All rights reserved.

Trace file: ora_26971.trc

Sort options: prsela exeela fchela

********************************************************************************

count = number of times OCI procedure was executed

cpu = cpu time in seconds executing

elapsed = elapsed time in seconds executing

disk = number of physical reads of buffers from disk

query = number of buffers gotten for consistent read

current = number of buffers gotten in current mode (usually for update)

rows = number of rows processed by the fetch or execute call

********************************************************************************

SELECT A.EMPLID

,A.EMPL_RCD#

FROM PS_JOB A

,PS_PERSONAL_DATA P




22







AND F.EFFDT>A.EFFDT












AND L.COMPANY='046'




ORDER BY A.EMPLID ASC

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------

Parse 1 0.79 0.89 3 0 64 0

Execute 1 0.00 0.00 0 0 0 0

Fetch 1 1056.25 1253.05 102488 1661197 8 12

------- ------ -------- ---------- ---------- ---------- ---------- ----------

total 3 1057.04 1253.94 102491 1661197 72 12

23

Misses in library cache during parse: 1

Optimizer goal: FIRST_ROWS

Parsing user id: 269 (SYSADM)

Rows Execution Plan

------- ---------------------------------------------------

0 SELECT STATEMENT GOAL: FIRST_ROWS

12 SORT (ORDER BY)

0 CONCATENATION

753 FILTER

753 NESTED LOOPS

233376 TABLE ACCESS GOAL: ANALYZED (FULL) OF 'PS_JOB'

753 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF

'PS_PERSONAL_DATA'

1506 INDEX GOAL: ANALYZED (RANGE SCAN) OF

'PS_PERSONAL_DATA' (NON-UNIQUE)

32361 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'PS_PAY_LINE'

106661796 INDEX GOAL: ANALYZED (RANGE SCAN) OF 'PS#PAY_LINE'

(NON-UNIQUE)

22090 SORT (AGGREGATE)

38167 INDEX GOAL: ANALYZED (RANGE SCAN) OF 'PSAJOB'

(NON-UNIQUE)

12632 FILTER

12632 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'PS_JOB'


(NON-UNIQUE)

0 SORT (AGGREGATE)


(NON-UNIQUE)

0 FILTER

24

0 NESTED LOOPS



'PS_PERSONAL_DATA'





(NON-UNIQUE)



(NON-UNIQUE)

12632 FILTER



(NON-UNIQUE)

0 SORT (AGGREGATE)


(NON-UNIQUE)

1422 FILTER

1422 NESTED LOOPS



'PS_PERSONAL_DATA'





(NON-UNIQUE)



25

(NON-UNIQUE)

12632 FILTER



(NON-UNIQUE)

0 SORT (AGGREGATE)


(NON-UNIQUE)

47360 FILTER

47360 NESTED LOOPS



'PS_PERSONAL_DATA'





(NON-UNIQUE)



(NON-UNIQUE)

12632 FILTER



(NON-UNIQUE)

0 SORT (AGGREGATE)


(NON-UNIQUE)

************************************************************************

26

The Tkprof output made it readily apparent that the index scan of PS#PAY_LINE was tremendously expensive. The major problem was the number of blocks being accessed to get the PS_PAY_LINE data.

Given that information, efforts re-focussed on access to PS_PAY_LINE. The CALC query tries to access the following fields:

emplid, empl_rcd#, company, paygroup, pay_end_dt, and off_cycle.

The PS#PAY_LINE index being used was significantly wider than needed for this query. So an index targeted specifically for the CALC query was created:

Figure 12: PS_PAY_LINE1_NDX

create index PS_PAY_LINE1_NDX on PAY_PAY_LINE(emplid, empl_rcd#, company, paygroup,

pay_end_dt, off_cycle);

analyze index PS_PAY_LINE1_NDX estimate statistics 30 percent;

The CALC query was then tested against the system. The results were fabulous. The CALC query finished in 1:00.51 minutes.

Figure 13: Final Tkprof for CALC

TKPROF: Release 7.3.4.0.0 - Production on Thu Dec 2 11:23:53 1999

Copyright (c) Oracle Corporation 1979, 1996. All rights reserved.

Trace file: ora_28091.trc

Sort options: prsela exeela fchela

********************************************************************************

count = number of times OCI procedure was executed

cpu = cpu time in seconds executing

elapsed = elapsed time in seconds executing

disk = number of physical reads of buffers from disk

query = number of buffers gotten for consistent read

current = number of buffers gotten in current mode (usually for update)

rows = number of rows processed by the fetch or execute call

********************************************************************************

SELECT A.EMPLID

27

,A.EMPL_RCD#

FROM PS_JOB A

,PS_PERSONAL_DATA P










AND F.EFFDT>A.EFFDT












AND L.COMPANY='046'




ORDER BY A.EMPLID ASC

call count cpu elapsed disk query current rows

28

------- ------ -------- ---------- ---------- ---------- ---------- ----------

Parse 1 0.07 0.07 0 0 20 0

Execute 1 0.00 0.00 0 0 0 0

Fetch 1 56.69 87.91 106106 493984 0 12

------- ------ -------- ---------- ---------- ---------- ---------- ----------

total 3 56.76 87.98 106106 493984 20 12

Misses in library cache during parse: 1

Optimizer goal: FIRST_ROWS

Parsing user id: 269 (SYSADM)

Rows Execution Plan

------- ---------------------------------------------------

0 SELECT STATEMENT GOAL: FIRST_ROWS

12 SORT (ORDER BY)

49535 FILTER

49535 NESTED LOOPS


53695 BITMAP CONVERSION (TO ROWIDS)

12 BITMAP INDEX (SINGLE VALUE) OF 'PS_JOB1_NDX'


'PS_PERSONAL_DATA'



47944 INDEX GOAL: ANALYZED (RANGE SCAN) OF 'PS_PAY_LINE1_NDX'

(NON-UNIQUE)



(NON-UNIQUE)

12632 FILTER


29


(NON-UNIQUE)

0 SORT (AGGREGATE)


(NON-UNIQUE)

************************************************************************

The number of blocks processed using the new index were significantly fewer than in the original query.

6.3 Final Results

The batch testing was all performed against a copy of the production database. The copy had a far smaller SGA than the production database. When the new indexes were implemented in production the CALC query timings were even better than on the test database. The CALC query completed in 12 seconds on the production database. As a result, the full calc process performed considerably better. Formerly, the calc process required 3 hours 14 minutes to complete. The new indexes decreased that time to 2 hours 9 minutes. The confirm process, which runs similar queries, ran in 2 hours less time.

7 Decision Support System


The DSS study was conducted by Raman Batra. Raman’s database environment is mixed decision support, OLTP, and batch processing. Payroll is run several times a week to support different sites and companies. Analysts in payroll and benefits generate reports depending upon the requirements of company management and functional areas. The analysts use PSQUERY to generate adhoc queries. The queries are completely unpredictable. Their occurrence is random and driven by business needs.

Some of the queries run for hours, and in a few cases they run overnight. More importantly, the resources consumed by these queries overload the database server during OLTP and Payroll runs. It was very common to have a few big queries take 40-60% CPU of a 4-way 400 MHz Solaris E5000 Server. Since the queries constantly change, it is very hard to tune for them. There is not much value in creating indexes and attempting to tune SQL unless there is a specific query which is run repeatedly.

30

Initially, Raman examined the top ten processes (CPU and memory) to identify problem queries. Sometimes an analyst would call the DBA asking for help on a specific query. Addressing the issue on a case-by-case basis provided little relief. The queries continued to change which constantly caused new problems. A more general solution was needed.

It was obvious the situation was crying for intelligent optimizer choices. Most analysts do not want to be bothered with join orders and careful planning of SQL statements. They are concerned with coding business logic. However, it was impossible to implement the CBO for the system without going through a significant stage of testing and planning. An interim solution was required.

7.2 The Infamous Security Tree

Raman began to collect offensive SQL statements to maintain a history and find some common trends. A detailed analysis of the SQL generated by the adhoc queries and reports found that the Peoplesoft security views were a common factor which could be improved. Virtually all queries generated by PSQUERY will automatically have the security views joined into the query by PSQUERY. This join is not under the control of the users. In Peoplesoft parlance, the result is called a search record. PS_PERSONAL_SRCH_Q is an example of such a view. It was obvious that improving the performance of the security tree would help overall performance of queries via PSQUERY.

7.3 Case Studies

In order to improve the performance of the views, Raman experimented with a number of approaches. Custom indexes were of limited use due to the views being joined with a larger query. A different approach was needed, so Raman began experimenting with views. The FIRST_ROWS hint was added to 7 views. They also reviewed the ORDERED , USE_MERGE and USE_HASH hints, however, the FIRST_ROWS hint seemed to need less work and gave better results.

The hints yielded a dramatic improvement in performance. Customers started to ask, “what did you do ?” A query nicknamed "The Bible", which formerly took almost an hour, now it ran in less than a minute. Benchmarks tests with the twelve "baddest and meanest" queries were very encouraging. The test result were so convincing that the hints were moved into Production.

Sure, putting hints in view is a customization. But the gains were dramatic. Also, putting hints in views can be done through Peopletools, so they can be carried forward during upgrades.

As an added bonus, once the FIRST_ROWS hint is put in the view syntax, any query using the VIEW in a join statement automatically invokes the Cost-Based Optimizer. We were therefore able to get the advantages of the CBO, but limit it to the PSQUERY tool. Batch jobs never use the security views. As a result, they never invoke the cost based optimizer.

There is one important caveat to this approach. You MUST run statistics on the database regularly for this to work properly. The CBO needs statistics to work well. If the statistics are not present, it will guess. That can be a very bad thing.

31

Figure 14: Run-time comparison of 12 queries between Rule-Based Optimizer and Cost-Based Optimizer.

Figure 15:PS_PERSONAL_SRCH View

CREATE OR REPLACE VIEW SYSADM.PS_PERSONAL_SRCH

(

EMPLID, OPRCLASS, EMPL_RCD#, NAME, LAST_NAME_SRCH, SSN,

SIN, NATIONAL_ID, ACCESS_CD

)

AS

SELECT /*+ FIRST_ROWS */

B.EMPLID ,C.OPRID ,B.EMPL_RCD# ,A.NAME ,A.LAST_NAME_SRCH ,A.SSN ,A.SIN ,A.NATIONAL_

ID ,C.ACCESS_CD

FROM PS_JOB B ,

PS_PERSONAL_DATA A ,

PS_SCRTY_TBL_SUPRV C

WHERE

------------- (SQL has been truncated here)

-------------C.K_HOURLY_SALARY = B.EMPL_TYPE

);

32

Figure 16: Details of Query2 (without hinted PS_PERSONAL_SRCH_Q View)

SELECT A.EMPLID, PS_RAM_NAME_LOOKUP(A.EMPLID), E.SSN, A.COMPANY,

B.DESCR, C.STATE, A.WORKERS_COMP_CD, D.DESCR, A.PAYGROUP, F.DESCR, G.ERNCD,

H.DESCR, G.BALANCE_YEAR, G.BALANCE_PERIOD, G.GRS_MTD, B.COMPANY,

TO_CHAR(B.EFFDT,'YYYY-MM-DD'),F.COMPANY,F.PAYGROUP,TO_CHAR(F.EFFDT,'YYYY-MM-DD'),

H.ERNCD,TO_CHAR(H.EFFDT,'YYYY-MM-DD')

FROM PS_K_JOB_QUERY_VW A,

PS_PERSONAL_SRCH_Q A1,

PS_COMPANY_TBL B, PS_LOCATION_TBL C,

PS_K_CO_WCOMP_TBL D, PS_PERSONAL_DATA E,

PS_PAYGROUP_TBL F, PS_EARNINGS_BAL G, PS_EARNINGS_TBL H

WHERE A.EMPLID IN

('00156','00871','00959','01203','01706','01749','01915','02316','02475','02628','02967','

03403','03426','03476','03583','04267','05130','05423','06107','06108','06303','06576','06

771','07252','07282','07422','07463','08380','09165','09340','09373','09442','09531','0992

6','10576','11496','11886','12028','12038','12336','12530','13050','14336','14471','14972'

,'16956','18069','18070','18362','18442','18816','19243','19762','19857','19858','20560','

21517','21687','21791','22589','22718','22910','23097','23936','23960','24033','24046','24

108','24115','24168','24308','24404','24444','24768','24921','25028','25137','25171','2546

9','25480','26239','26980','27160','27217','27218','27410','27569','27729','28065','29750'

,'30950','31508','31960','32249','32470','32825','32952','33803','33835','37430','37865','

38266','38543','45433','49184','54100','54840','63550','68060','68690','70137','71818','75

420','76390','78090','80500','81220','83696','83985','89930','93892','97288')

AND B.COMPANY = A.COMPANY

AND C.LOCATION = A.LOCATION

AND D.WORKERS_COMP_CD = A.WORKERS_COMP_CD

AND D.COMPANY = A.COMPANY

AND E.EMPLID = A.EMPLID

AND F.PAYGROUP = A.PAYGROUP

AND F.COMPANY = A.COMPANY

AND H.ERNCD = G.ERNCD

AND A.EMPLID = A1.EMPLID

AND A.EMPL_RCD# = A1.EMPL_RCD#

33

AND A1.OPRID = 'BATRAR'

AND E.EMPLID = A1.EMPLID

AND G.EMPLID = A.EMPLID

AND G.BALANCE_YEAR IN (1998,1999)

AND G.BALANCE_PERIOD IN (1,2,3,8,9,10,11,12)

AND G.BALANCE_PERIOD = (SELECT MAX(BALANCE_PERIOD) FROM PS_EARNINGS_BAL GG

WHERE GG.COMPANY=G.COMPANY

AND GG.ERNCD=G.ERNCD

AND GG.EMPLID= G.EMPLID

AND GG.BALANCE_ID=G.BALANCE_ID

AND GG.BALANCE_YEAR=G.BALANCE_YEAR)

AND ( A.EFFDT =

(SELECT MAX(EFFDT) FROM PS_K_JOB_QUERY_VW

WHERE A.EMPLID = EMPLID

AND A.EMPL_RCD# = EMPL_RCD#

AND EFFDT <= SYSDATE)

AND A.EFFSEQ =

(SELECT MAX(EFFSEQ) FROM PS_K_JOB_QUERY_VW



AND A.EFFDT = EFFDT)

AND B.EFFDT =

(SELECT MAX(EFFDT) FROM PS_COMPANY_TBL

WHERE B.COMPANY = COMPANY

AND EFFDT <= A.EFFDT)

AND C.EFFDT =

(SELECT MAX(EFFDT) FROM PS_LOCATION_TBL

WHERE C.LOCATION = LOCATION


AND D.EFFDT =

(SELECT MAX(EFFDT) FROM PS_K_CO_WCOMP_TBL

34

WHERE D.COMPANY = COMPANY

AND D.WORKERS_COMP_CD = WORKERS_COMP_CD


AND F.EFFDT =

(SELECT MAX(EFFDT) FROM PS_PAYGROUP_TBL

WHERE F.COMPANY = COMPANY

AND F.PAYGROUP = PAYGROUP


AND H.EFFDT =

(SELECT MAX(EFFDT) FROM PS_EARNINGS_TBL

WHERE H.ERNCD = ERNCD


)

Execution Plan for the above query:

Query Plan

----------


CONCATENATION

FILTER

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

35

TABLE ACCESS BY INDEX ROWID

PS_SCRTY_TBL_SUPRV

INDEX RANGE SCAN

PSASCRTY_TBL_SUPRV




PS_PERSONAL_DATA


TABLE ACCESS BY INDEX ROWID PS_PAYGROUP_TBL

INDEX RANGE SCAN PS#PAYGROUP_TBL

TABLE ACCESS BY INDEX ROWID PS_K_CO_WCOMP_TBL

INDEX RANGE SCAN PS_K_CO_WCOMP_TBL




TABLE ACCESS BY INDEX ROWID PS_EARNINGS_BAL

INDEX RANGE SCAN PSBEARNINGS_BAL

TABLE ACCESS BY INDEX ROWID PS_EARNINGS_TBL

INDEX RANGE SCAN PS#EARNINGS_TBL

TABLE ACCESS BY INDEX ROWID PS_LOCATION_TBL

INDEX RANGE SCAN PS#LOCATION_TBL

TABLE ACCESS BY INDEX ROWID PS_COMPANY_TBL

INDEX RANGE SCAN PS#COMPANY_TBL

SORT AGGREGATE

INDEX RANGE SCAN PS_EARNINGS_BAL

SORT AGGREGATE

INDEX RANGE SCAN PSDJOB

SORT AGGREGATE



SORT AGGREGATE

36

INDEX RANGE SCAN PS_COMPANY_TBL

SORT AGGREGATE

INDEX RANGE SCAN PS_LOCATION_TBL

SORT AGGREGATE


SORT AGGREGATE

INDEX RANGE SCAN PS_PAYGROUP_TBL

SORT AGGREGATE

INDEX RANGE SCAN PS_EARNINGS_TBL

SORT AGGREGATE


SORT AGGREGATE



FILTER

TABLE ACCESS BY INDEX ROWID PSTREENODE

INDEX RANGE SCAN PSCPSTREENODE

INDEX RANGE SCAN PS#SCRTY_TBL_SUPRV

FILTER

SORT AGGREGATE


SORT AGGREGATE


SORT AGGREGATE



SORT AGGREGATE


SORT AGGREGATE


SORT AGGREGATE

37


SORT AGGREGATE


SORT AGGREGATE


SORT AGGREGATE


SORT AGGREGATE



FILTER




NESTED LOOPS


INDEX RANGE SCAN PS#COMPANY_TBL

NESTED LOOPS


INDEX RANGE SCAN PS#LOCATION_TBL

NESTED LOOPS


INDEX RANGE SCAN PS#EARNINGS_TBL

NESTED LOOPS


INDEX RANGE SCAN PSBEARNINGS_BAL

NESTED LOOPS


NESTED LOOPS



38

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS


PS_SCRTY_TBL_SUPRV

INDEX RANGE SCAN

PSASCRTY_TBL_SUPRV




PS_PERSONAL_DATA






Note: This plan is truncated for printing here since it repeats itself for every employee in the IN clause. The complete execution plan, if printed goes onto about 5-6 pages. The RULE based optimizer is not able to handle an IN clause too well.

Figure 17: Details of Query2 (with hinted PS_PERSONAL_SRCH_Q view)

SELECT A.EMPLID, PS_RAM_NAME_LOOKUP(A.EMPLID), E.SSN, A.COMPANY,

B.DESCR, C.STATE, A.WORKERS_COMP_CD, D.DESCR, A.PAYGROUP, F.DESCR, G.ERNCD,

H.DESCR, G.BALANCE_YEAR, G.BALANCE_PERIOD, G.GRS_MTD, B.COMPANY,

TO_CHAR(B.EFFDT,'YYYY-MM-DD'),F.COMPANY,F.PAYGROUP,TO_CHAR(F.EFFDT,'YYYY-MM-DD'),

H.ERNCD,TO_CHAR(H.EFFDT,'YYYY-MM-DD')

FROM PS_K_JOB_QUERY_VW A,

PS_PERSONAL_SRCH_Q A1,

PS_COMPANY_TBL B, PS_LOCATION_TBL C,

39

PS_K_CO_WCOMP_TBL D, PS_PERSONAL_DATA E,

PS_PAYGROUP_TBL F, PS_EARNINGS_BAL G, PS_EARNINGS_TBL H

WHERE WHERE A.EMPLID IN

('00156','00871','00959','01203','01706','01749','01915','02316','02475','02628','02967','

03403','03426','03476','03583','04267','05130','05423','06107','06108','06303','06576','06

771','07252','07282','07422','07463','08380','09165','09340','09373','09442','09531','0992

6','10576','11496','11886','12028','12038','12336','12530','13050','14336','14471','14972'

,'16956','18069','18070','18362','18442','18816','19243','19762','19857','19858','20560','

21517','21687','21791','22589','22718','22910','23097','23936','23960','24033','24046','24

108','24115','24168','24308','24404','24444','24768','24921','25028','25137','25171','2546

9','25480','26239','26980','27160','27217','27218','27410','27569','27729','28065','29750'

,'30950','31508','31960','32249','32470','32825','32952','33803','33835','37430','37865','

38266','38543','45433','49184','54100','54840','63550','68060','68690','70137','71818','75

420','76390','78090','80500','81220','83696','83985','89930','93892','97288')

AND B.COMPANY = A.COMPANY

AND C.LOCATION = A.LOCATION

AND D.WORKERS_COMP_CD = A.WORKERS_COMP_CD

AND D.COMPANY = A.COMPANY

AND E.EMPLID = A.EMPLID

AND F.PAYGROUP = A.PAYGROUP

AND F.COMPANY = A.COMPANY

AND H.ERNCD = G.ERNCD

AND A.EMPLID = A1.EMPLID

AND A.EMPL_RCD# = A1.EMPL_RCD#

AND A1.OPRID = 'BATRAR'

AND E.EMPLID = A1.EMPLID

AND G.EMPLID = A.EMPLID

AND G.BALANCE_YEAR IN (1998,1999)

AND G.BALANCE_PERIOD IN (1,2,3,8,9,10,11,12)

AND G.BALANCE_PERIOD = (SELECT MAX(BALANCE_PERIOD) FROM PS_EARNINGS_BAL GG

WHERE GG.COMPANY=G.COMPANY

AND GG.ERNCD=G.ERNCD

AND GG.EMPLID= G.EMPLID

AND GG.BALANCE_ID=G.BALANCE_ID

40

AND GG.BALANCE_YEAR=G.BALANCE_YEAR)

AND ( A.EFFDT =

(SELECT MAX(EFFDT) FROM PS_K_JOB_QUERY_VW




AND A.EFFSEQ =

(SELECT MAX(EFFSEQ) FROM PS_K_JOB_QUERY_VW



AND A.EFFDT = EFFDT)

AND B.EFFDT =

(SELECT MAX(EFFDT) FROM PS_COMPANY_TBL

WHERE B.COMPANY = COMPANY


AND C.EFFDT =

(SELECT MAX(EFFDT) FROM PS_LOCATION_TBL

WHERE C.LOCATION = LOCATION


AND D.EFFDT =

(SELECT MAX(EFFDT) FROM PS_K_CO_WCOMP_TBL

WHERE D.COMPANY = COMPANY

AND D.WORKERS_COMP_CD = WORKERS_COMP_CD


AND F.EFFDT =

(SELECT MAX(EFFDT) FROM PS_PAYGROUP_TBL

WHERE F.COMPANY = COMPANY

AND F.PAYGROUP = PAYGROUP


AND H.EFFDT =

(SELECT MAX(EFFDT) FROM PS_EARNINGS_TBL

41

WHERE H.ERNCD = ERNCD


)

Explain Plan for the above SQL:

Query Plan

----------


FILTER

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

NESTED LOOPS

INLIST ITERATOR CONCATENATED



SORT AGGREGATE




SORT AGGREGATE

INDEX RANGE SCAN

PS_COMPANY_TBL


42


SORT AGGREGATE



INDEX RANGE SCAN PS#K_CO_WCOMP_TBL





SORT AGGREGATE




SORT AGGREGATE







SORT AGGREGATE


TABLE ACCESS BY INDEX ROWID PS_SCRTY_TBL_SUPRV

INDEX RANGE SCAN PSASCRTY_TBL_SUPRV

FILTER




SORT AGGREGATE


SORT AGGREGATE

43

TABLE ACCESS FULL PS_K_CO_WCOMP_TBL

SORT AGGREGATE


SORT AGGREGATE


Note: The IN statement is handled very well by the Cost-Based Optimizer and the INLIST Iteration works wonders .

7.4 Final Results

The results were excellent. All adhoc queries referencing the 7 views started to use the Cost Based Optimizer and run-times improved across the board. A total of 7 views were modified with the same hint. At the time of this writing, Raman is working on tuning the payroll jobs. Once that is complete, he plans to implement the Cost-Based Optimizer for the instance rather than just hinting the views.

8 Conclusions and Summary

Implementing the Oracle Cost Based Optimizer can be of significant benefit to Peoplesoft systems. The first_rows method will generally provide significant performance improvements for online users. In addition, with careful tuning and indexing, the results of batch and DSS jobs can also experience great improvements. Finally, one must realize that the CBO is the way of the future. Oracle has set a clear path toward making the CBO the heart of future performance enhancements for its’ flagship database product. As such, it will be a minimum requirement that systems implement the CBO in order to reap the benefits of improvements in the Oracle RDBMS.

44

9 Appendix A: Scripts

-- allActive.sql

-- This routine will produce a list

-- of all active users currently connected to the

-- system. The list will indicate various

-- things including the status of the connection,

-- the type of command currently being executed,

-- and whether the session is blocked by another

-- user.

-- author James E. Strange

--

set timing off

set pause off

set pagesize 9000

set linesize 100

column sid heading SID format 999

column spid heading OSPID format 999999

column username heading USER format A10

column osuser heading OSUSER format A10

column machine heading MACHINE format A22

column status heading STATUS format A8

column name heading COMMAND format A15

column lockwait format A8

select s.sid, proc.spid, s.username, s.osuser,

s.machine, s.status, a.name, nvl(s.lockwait, 'NO') BLOCKED

from v$session s, audit_actions a,

v$process proc

where a.action = s.command

45

and proc.addr = s.paddr

and s.username is not null

and s.status = 'ACTIVE'

order by s.status, s.sid

/

-- sql.sql

-- This routine lists the sql for a selected

-- sid. The second second query will indicate

-- the optimizer mode and the number of rows

-- processed by the query.

--


--

select text.sql_text

from

v$sqltext_with_newlines text, v$session sess

where

text.address = sess.sql_address and

text.hash_value = sess.sql_hash_value and

sess.sid = &sid

order by piece

/

-- checkLatches.sql

-- Author: James E. Strange

-- This script displays the status of latches. It looks at them from 3 different ways.

-- The first section simply lists the latch statistics.

-- The second section looks at the willing-to-wait stats.

-- The last section looks at the not-willing-to-wait point of view.

46

--

set pagesize 900

set linesize 120

column name format A40

select l.latch# latch#, l.misses misses, l.gets gets, n.name name from v$latch l,

v$latchname n where n.latch# = l.latch#

order by name

/

select name latch_name, gets, misses,

round((gets-misses)/decode(gets,0,1,gets),3) hit_ratio,

sleeps, round(sleeps/decode(misses,0,1,misses),3)

"SLEEPS/MISS"

from v$latch

where gets != 0

order by name

/

select name latch_name, immediate_gets nowait_gets,

immediate_misses nowait_miss,

round((immediate_gets/(immediate_gets+immediate_misses)),3) nowait_hit_ratio

from v$latch

where immediate_gets + immediate_misses != 0

order by name

/

-- makeExplain.sql

-- This routine lists the sql for a selected

-- sid into a file. The output is formatted to

-- make it relatively simple to create a file to

-- use for getting an explain plan. This script

47

-- assumes the file name should be of the form 'test#'

-- where the # is a number. The script will prompt

-- for a number to substitute for the #.

--


--

set echo off

set heading off

set linesize 64

spool test.sql

select 'explain plan set statement_id = ''test&1'' for ' from dual;

select

rtrim(text.sql_text)

from

v$sqltext_with_newlines text, v$session sess

where

text.address = sess.sql_address and

text.hash_value = sess.sql_hash_value and

sess.sid = &sid

order by piece

;

select ';' from dual;

spool off

48

10 Appendix B: New Indexes

What follows is a partial list of the custom indexes added to Peoplesoft at Jim’s employer in order to support various batch and reporting jobs:

TABLE_NAME INDEX_NAME COLUMN_NAME

------------------------------ ------------------ ----------------------

PS_EMPLOYEES PS_EMPLOYEES1_NDX GL_PAY_TYPE

PS_EMPLOYEES EMPL_STATUS

PS_EMPLOYEES EMPLID

PS_EMPLOYEES EMPL_RCD#

PS_EMPLOYEES EFFDT

PS_EMPLOYEES EFFSEQ

PS_JOB PS_JOB1_NDX COMPANY

PS_JOB PAYGROUP

PS_JOB PS_JOB2_NDX GL_PAY_TYPE

PS_JOB EMPLID

PS_JOB GRADE

PS_JOB SAL_ADMIN_PLAN

PS_JOB POSITION_NBR

PS_JOB EMPL_RCD#

PS_JOB EFFDT

PS_PAY_DEDUCTION PS_PAY_DEDUCTION1_NDX DEDCD

PS_PAY_LINE PS_PAY_LINE1_NDX EMPLID

PS_PAY_LINE EMPL_RCD#

PS_PAY_LINE COMPANY

PS_PAY_LINE PAYGROUP

PS_PAY_LINE PAY_END_DT

PS_PAY_LINE OFF_CYCLE

49

PS_PAY_LINE PS_PAY_LINE2_NDX EMPL_RCD#

PS_PAY_LINE EMPLID

PS_PAY_LINE COMPANY

PS_PERSONAL_DATA PS_PERSONAL_DATA1_NDX PER_STATUS

PS_PERSONAL_DATA PER_TYPE

PS_PERSONAL_DATA EMPLID

PS_W2_EE PS_W2_EE1_NDX EMPLID

PS_W2_EE PROCESS_FLAG

11 Appendix C: Test Environments

Production Online Environment for Batch and Online Case Studies

Database Server HPUX 10.20

T-600 system 6 CPUs

3.5 Gigabytes of RAM

Oracle 7.3.4.2

SGA = 500MB

Optimizer Mode First_Rows

Client Machine Compaq Armada

333 MZ CPU 96 MB RAM

Application Server Co-resident on database server

Database Size 30 Gig – 17000 employees

PS Applications HR, Payroll, Benefits, Time and Labor, Self Service

PS Version 7.02

Production Batch Environment for DSS Case Study

Database Server Sun Solaris 2.5.1

E5000 4 CPUs

3.0 Gigabytes of RAM

50

Oracle 8.0.5

SGA = 680MB

Optimizer Mode Rule

Client Machine Dell Laptop

300 - II MZ CPU 128 MB RAM

Application Server Tuxedo, Logical three-tier, Co-resident on database server

Database Size 26 Gig – approx. 13000 employees

PS Applications HR, Payroll, Benefits

PS Version 7.03

12 Appendix D: Recommended References

1. "Tuning Oracle for PeopleSoft Applications", Jacob C. Thomas, IOUG(A)-Live 1998, Orlando, Florida.

2. Oracle Performance Tuning, Second Edition, Gurry and Corrigan, O'Reilly Books (1996)

3. Oracle 8 Server Tuning Manual, Oracle 8 Documentation, Oracle Corporation (June 1998)

4. Oracle Performance Tuning Tips and Techniques, Niemic Rich et. al, Oracle Press,McGraw Hill (199

51

2000-01-28 strange - perf. for peoplesoft apps

Documents

peoplesoft applications

peoplesoft hr applications

oracle database administrators

rulebased optimizer

oracle optimizer54

oracle community

optimizer modes

database environments