configuring peoplesoft global payroll for optimal performance session 507...
TRANSCRIPT
Configuring PeopleSoft
Global Payroll for Optimal
PerformanceSession 507
www.go-faster.co.uk
m
www.omniasolutions.com
2
Who are we?
David Kurtz
• Independent Consultant working for UBS Swiss GP project
• Systems Performance Tuning Oracle Databases Unix PeopleSoft Applications
Gene Pirogovsky
• Independent Consultant working for UBS Swiss GP project
• Global Payroll
• Interfaces
• Customizations
3
Configuring Global Payroll
Physical Database Considerations
• Oracle specific
• Reducing I/O
• CPU overhead
GP Changes
• Reduce CPU consumption of rules Engine
• Data Migration
4
Initial Impressions
Payroll is calculated by a Cobol program•GPPDPRUN
•Single non-threaded process
•Four Stages Cancel Identify (Re-)Calculate Finalise
5
Two stages with different behaviours
Identify• Populating temporary
work tables
• Opening cursors
• Database Intensive
• ~20 minutes
Calculation• Evaluation of rules
(Cobol only)
• Batch insert of results into database
• Cobol (CPU) Intensive
• ~5000 segments / hour / stream (was 1200)
6
What is Streaming?
Employees are split into groups defined by ranges of employee ID
Each group/range can be processed by a different instance/stream of GPPDPRUN
The streams can then be run in parallel.
Vanilla PeopleSoft functionality.
7
Why is Streaming Necessary? GPPDPRUN is a standard Cobol program. It is a single threaded process One Cobol process can only run on one
CPU at any one time
33000 employees at 5000 employees/hour/stream•6.6hrs if run in one stream•27.5 hours at 1200/hr
On a multi-processor server streaming enables consumption of extra CPU.
8
Calculation of Stream Definitions
Objective is roughly equal processing time for all stream•PS_GP_PYE_PRC_STAT indicates work to
be done by payroll.•Calculate ranges of roughly equal
numbers of rows for this table•Script using Oracle’s Analytic functions
that directly populates PS_GP_STRM
This does NOT lead to equally sized GP_RSLT* tables.
9
Partition Boundary Creep
As new employees hired EMPLIDs allocated into the same stream.•That stream starts to run longer.•Effective execution time is maximum
execution time for all streams.
•Need to periodically recalculate stream ranges
•Need to reflect this is physical changes.
10
Database Contention
Rollback Contention
Snapshot Too Old
Insert Contention
I/O Volume•Datafile I/O•Redo/Archive Log Activity
11
Rollback Contention
Working Storage Tables•Shared by all streams•Rows inserted/deleted during run•Different Streams never create locks that
block each other,•Do update different rows in same block
during processing•1 interested transaction per stream in
many blocks.•There is a additional rollback overhead of
16 bytes per row if two rows in same block -v- different blocks
updates of ~<100 bytes / row
12
Read Consistency
Oracle guarantees that data is consistent throughout life of a query• If a block has been updated by another
transaction since a long running query started, it must be possible to reconstruct the state of that block at the time the query started using the rollback segment.
• If that information cannot be found in the rollback segment the long running query fails with ORA-01555.
13
ORA-01555 Snapshot Too Old Rollback segments are not extended
for read consistency. Additional rollback overhead can cause
rollback segments to spin and wrap. Error message also described a
‘rollback segments too small.’
In this case, to simply extend the segments is the wrong response.
CPU overhead to navigate rollback segment header chain
14
Insert Contention
During the calculation phase results are written to the result tables.
A number of stream can simultaneously insert into the same result tables.
Increases chance that one block will contain rows relating to more than one stream.
This in turn causes rollback problems during the cancel in the next calculation.
15
Another cause of ORA-1555• If not processing calendar for the first time,
previous results cancelled Result table are deleted Monolithic deletes from each table.
• If Streams start together tend to delete same table at same time in each stream.
•A long running delete is also a query for the purposes of read consistency. It is necessary to reconstruct a block as at the time the long running delete started in order to delete a row from it.
Reconstruction occurs during ‘consistent read’. Deletes by primary key columns, thus Oracle tends to look each row up row by index. Thus index reads also ‘consistent’.
16
Datafile and Log Physical I/O Activity
During the identify phase data is shuffled from table to table• This generates datafile and redo log I/O
Rollback activity is also written to disk, undo information is also written to the redo log.
All the data placed in the temporary working tables by a stream is of no use to any other instance of the calculation process.
It will be deleted by a future process.
17
High Water Marks
The working storage tables tend to be used to drive processing.
Thus, the SQL tends to use full table scans.
In Oracle, High Water Mark is the highest block that has ever contained data.
Full Scans scan the table up to the high water mark.
Temporary tables contain data for ALL streams.
All streams can have to scan data for all streams.
18
How to avoid inter-stream contention?
Keep rows from different streams in different blocks
Each block should contain rows for one and only one stream.
Two Oracle Features
19
What is Partitioning?
Logically,•a partitioned table is a still a single table
Physically,•each partition is a separate table.• in a range partitioned table, the partition
in which a row is placed is determined by the value of one or more columns.
Local Index• is partitioned on the same logical basis as
the table.
20
How should Partitioning used in GP?
Largest Result tables range each partitioned on EMPLID to match GP streaming• 1 stream : 1 partition
• Thus each stream references one partition in each result table.
• Only 1 interested transaction per block
• Indexes ‘locally’ partitioned
Partitioning really designed for DSS systems. Only efficient for large tables.• GP_RSLT_ACUM, GP_RSLT_ERN_DED,
• GP_RSLT_PIN, GP_RSLT_PI_DATA
• GP_PYE_PRC_STAT, GP_PYE_SEG_STAT
21
Global Temporary Tables
Definition is permanently defined in database catalogue.
Physically created on demand by database in temporary tablespace for duration of session/transaction. Then dropped.
Each session has its own copy of each referenced GT table.
Each physical instance of each GT table only contains data for one stream.
Working Storage Tables PS_GP_%_WRK converted to GT tables.
22
Global Temporary Tables
Advantages• Not recoverable,
therefore no Redo/Archive Logging some undo information
improved performance
reduce rollback
• No High Water Mark problems
• Smaller object to scan.
• No permanent tablespace overhead.
Disadvantages• Does consume
temporary tablespace but only during payroll
• No CBO Statistics
• Can hamper debugging
• New in Oracle 8.1, some bugs.
23
How many streams should be run?
Cobol run on database server• Either Cobol is active
or database is active
• No more than one stream per CPU
• Perhaps CPUs -1 be careful not to starve database of CPU
run process scheduler at lower OS priority
Cobol and database on different servers• Cobol active for 2/3
of execution time.
• Up to 1.5 streams per CPU on Cobol server
• Up to 3 streams per CPU on database server
24
UBS Production Payroll Configuration
2 nodes•Database Node•Application Server/Process Scheduler Node
20 CPUs each
30 Streams•2/3 of 30 is 20, so all 20 application server
node CPUS active during calculate phase ‘nice’ the Cobol processes
•1/3 of 30 is 10, so 10 of 16 CPUs active important to leave some free CPU for database else spins escalated to sleeps generating latch contention
25
QA Payroll Configuration
2 nodes•Database Node•Application Server/Process Scheduler
Node 10 CPUs each
15 Streams•Full production volume payroll
•< 1 hour
26
Goals
How to create and test efficient rules that work
without adversely effecting performance
How best to identify problems particularly in the
area of system setup/data versus a problem in a rule
or underlying program
How to use GP payroll debugging tools
27
Efficient Rules
Responsible for two thirds of the execution time, and
so could produce the greatest saving, it will also
require the greatest effort.
Detailed functional and technical analysis of the
definition of the payroll rules.
28
Efficient Rules
The process ideally starts during the design stage
when various implementation schemes are analysed,
intermediate tests are performed and the most
efficient scheme is chosen.
All aspects of Global Payroll must be considered
since creating rules to simplify calculation can
adversely affect reporting or other online and batch areas and vice versa.
29
Efficient Rules
Arrays
Re-calculate?
Store / Don’t store
Formulas
Proration and Count
Historical rules
Generation control versus conditional section
30
Efficient Rules
Keyed by Employee - 1 select, multiple fetches, small result set to search
User Defined - 1 select, multiple fetches, all searches
in memory.
User Defined with the Reload Option - multiple selects, multiple fetches, small result set to search.
31
Efficient Rules
32
Efficient Rules
33
Efficient Rules
34
Efficient Rules
35
Efficient Rules
36
Efficient Rules
37
Efficient Rules
38
Efficient Rules
PIN_NUM PIN_NM Before AfterCount Time Average Count Time Average
1677 CH_CA_AR010 1569 2818.561.796405 1569 509.08 0.3244611679CH_CA_AR020 4379 2316.640.529034 4379 55.81 0.0127451269CH_CA_AR015 3138 1663.980.530268 3138 40.72 0.0129761986CH_FK_AR004 3138 1663.980.530268 3138 34.3 0.0109311521CH_CA_AR030 4610 115.520.025059 4610 38.11 0.0082671408CH_TX_AR005 704 111.430.158281 704 22.37 0.0317761912 CH_CA_AR012 1569 68.270.043512 1569 69.76 0.0444611206CH_TX_AR003 704 40.50.057528 704 7.88 0.011193
Total 8798.88 778.03Difference 2 hours 13 minutes 40 seconds running 1569
employees
39
Efficient Rules
After Modification - 8%
Before Modification -
92%
40
Migration/Customization
PI v. Array
•PI can be used during identification. •PI has special considerations during
eligibility checking. •PI allows easy override of components on
element definition such as Unit, Rate, Percent or Base.
•The Array cannot handle multiple instances of earning/deduction.
t126297:t126297:
41
Debugging Tools
Audit Trace
•Trace All •Trace Errors
•Large number of records, potential rollback segment size problems
•View on-line•Query with SQL
t126297:t126297:
42
Debugging Tools
t126297:t126297:
43
Debugging Tools
t126297:t126297:
44
Debugging Tools
select * from sysadm.ps_gp_audit_tbl
where emplid = '884324'
and cal_run_id = 'ErrMigr'
and pin_num = 40811
order by audit_sort_key, audit_seq_num;
select * from sysadm.ps_gp_audit_tbl
where emplid = '884324'
and cal_run_id = 'ErrMigr'
and audit_sort_key = 229
order by audit_seq_num;
t126297:t126297:
45
Debugging Toolsselect emplid, audit_sort_key as key,audit_seq_num as seq, pin_chain_rslt_num as rslt_num,b.pin_nm, a.pin_num,pin_status_ind as status, c.pin_nm,a.pin_parent_num as parent, a.fld_fmt as fmt,calc_rslt_val as num, date_pin_val as dateval,chr_pin_val as chr, pin_val_num as pin
from ps_gp_audit_tbl a, ps_gp_pin b, ps_gp_pin cwhere cal_run_id = 'U_22_CI0101' and (emplid, audit_sort_key) in (select emplid, audit_sort_key
from ps_gp_audit_tbl where cal_run_id = 'U_22_CI0101' and pin_num =
(select pin_num from ps_gp_pin where pin_nm = 'CH_EP_CHK_1002FF'))
and a.pin_num = b.pin_numand a.pin_parent_num = c.pin_numorder by emplid, audit_sort_key, audit_seq_num
t126297:t126297:
46
Debugging Tools
t126297:t126297:
47
Debugging ToolsAUDIT_SORT_KEY
AUDIT_SEQ_NUM
INSTANCE_NUM
SLICE_BGN_DT
SLICE_END_DT
PIN_CHAIN_LVL_NUM
PIN_CHAIN_RSLT_NUM
PIN_NUM
PIN_STATUS_IND
PIN_PARENT_NUM
FLD_FMT OLD_VALUE_IND
CALC_RSLT_VAL
CALC_ADJ_VAL
CALC_RAW_VAL
DATE_PIN_VAL
CHR_PIN_VAL
PIN_VAL_NUM
DIFF_SECONDS
BAD_TRACE_IND
SUM_INSTANCE_IND
48
Conclusion
Use of Partitioning and Global Temporary Tables reduce (almost eliminate) inter-stream contention.
This permits use of streaming to utilise all available CPUs.
GP will always be a CPU bound process•Rule Tuning will reduce CPU overhead• It is an on-going process
49
And there’s more
This has been a very concentrated session
Round Table Discussion session 626•15.30-16.30•Discuss some areas in more detail.
However, we do have time for some questions now...
Configuring PeopleSoft
Global Payroll for Optimal
PerformanceSession 507
www.go-faster.co.uk
m
www.omniasolutions.com