pbs analytics 12.2 chart calculations...

PBS Works is a division of

PBS AnalyticsTM 12.3

Chart Calculations

Altair PBS Analytics 12.3 Chart Calculations updated 3/28/14

Copyright© 2003-2014 Altair Engineering, Inc. All Rights Reserved

Altair PBS Works, Compute Manager™, Display Manager™, PBS™, PBS Works™, PBS Professional®, PBS Application Services™, PBS Analytics™, PBS Desktop™, PBS Por-tal™, e-BioChem™, e-Compute™ and e-Render™ are trademarks of Altair Engineering, Inc. and are protected under U.S. and international laws and treaties.

All other marks are the property of their respective owners.

Copyright notice does not imply publication. Contains trade secrets of Altair Engineering, Inc. Decompilation or disassembly of this software is strictly prohibited. This software is pro-tected under patent #6,859,792 and other patents pending. Usage of Altair Engineering, Inc. software is only as explicitly permitted as stated in the end user software license agreement.

Third Party Agreements

http://www.altairhyperworks.com/ClientCenter3rdPartyLicense.aspx

Table of Contents

PBS Analytics Chart Calculations1 Cluster Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1 Jobs Count by Day. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Jobs by Software by Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3 Jobs by Software by Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.4 Requested Vs Used Memory by Software. . . . . . . . . . . . . . . . . . . . 9

2 Cost Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1 Node Walltime by Group - Non Prime Time . . . . . . . . . . . . . . . . 122.2 Node Walltime by Group - Prime Time . . . . . . . . . . . . . . . . . . . . 14

3 Green Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1 Unused Node Capacity by Node . . . . . . . . . . . . . . . . . . . . . . . . . . 183.2 Unused Node Capacity by Quarter . . . . . . . . . . . . . . . . . . . . . . . . 203.3 Unused Vs Used Node Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.1 Average Wait Time Vs Average Overall Time by Software. . . . . 244.2 Average Wait Time by Software by User . . . . . . . . . . . . . . . . . . . 264.3 Jobs by Exit Status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.4 Successful Vs Unsuccessful Jobs by Node . . . . . . . . . . . . . . . . . . 284.5 Successful Vs Unsuccessful Jobs by Software . . . . . . . . . . . . . . . 304.6 User Job Efficiency and Productivity . . . . . . . . . . . . . . . . . . . . . . 32

PBS Analytics 12.3 Chart Calculations i

PBS Analytics Chart CalculationsThis document provides the calculation details for the default charts available in PBS Analyt-

ics ™ (PBSA) 12.3.

The default charts are divided into categories:

• Cluster Utilization

• Cost Analysis

• Green Computing

• Productivity

Sample data referenced in the chart calculation examples is provided for clarity, and does not necessarily correlate with the data displayed in the chart images.

Walltime calculations

When calculating node walltime, placement options other than “exclusive” (free:shared, none, and scatter) are considered as “non-exclusive”.

In order to resolve Spectrum ticket 296173 - “Node utilization exceeding 100% due to sus-pend/resume jobs”, the calculation of walltime has been changed to provide a workaround for this issue. When suspend/resume job information is written to the PBS Professional account-ing logs, a permanent solution will be implemented in a future release of PBSA. Please check the UCR attached to the Spectrum ticket 296173 for more details.

The following describes the new calculation for walltime (CPU walltime, memory walltime,

and node walltime):.

For all Total and Average walltime calculations, PBSA utilizes the value of the job’s resource_used.walltime, except for the following conditions:

• resources_used.walltime is negative.

• resources_used.walltime > ( end – start )

• running jobs where resources_used.walltime is not present

In these three cases, PBSA utilizes the job’s ( end - start ) to calculate walltime.

walltime is calculated as (end - start) except in the following scenario:

When resources_used.walltime < (end - start ), the value of walltime is calculated as a ratio of the walltime.

Proportionate Walltime Ratio = resources_used.walltime / (end – start )

PBSA walltime = (end – start) * Proportionate Walltime Ratio

PBS Analytics 12.3 Chart Calculations 1

PBS Analytics Chart Calculations

Data decimation

Data may appear to be missing from a default chart when viewing it via the web application. This can be due to:

• The chart property Auto Interval has been enabled for this chart. Auto Interval allows PBSA to set the axis scale and auto-generate the axis labels, therefore every value plotted on the chart is not represented as an axis label. You can disable Auto Interval, or you can adjust the view of the data using the scrollbar.

• Decimation is enabled by default. To technically improve the rendering of charts within the web browser, and to provide usable charts that show trends and patterns, PBSA uses a decimation or downsampling strategy to reduce the number of plot points displayed in a chart. Configurable limitations are set for both the x and y coordinates that essentially fil-ter the data that is plotted on the graph. By default, PBSA is configured to display at most 150 points on the x-axis, and 50 points on the y-axis in the case of stacked and xyz charts.

If you wish to disable data decimation, then you can do so by changing the configuration parameters. Please refer to section "Configuring plot point decimation" of the PBSA System Administrator's Guide.

2 PBS Analytics 12.3 Chart Calculations


1 Cluster Utilization

These group of charts provide historical data about the cluster such as cluster utilization, soft-ware usage, and job statistics. IT managers can use these charts to help understand usage, throughput, and availability of a site’s cluster over time.

The following charts are available within this category:

• Jobs count by day - shows the number of jobs present on the cluster

• Jobs by software by group - shows the software usage by custom group

• Jobs by software by node - shows the software usage by node

• Requested vs used memory by software - compares the average requested memory to theaverage used memory by software



1.1 Jobs Count by Day

This line chart shows the jobs that were present on the cluster. The following categories of jobs are counted:

• jobs that started and ended on that day

• jobs that started but did not end on that day

• jobs that ended on that day but did not start on that day

• jobs that were running but were deleted on that day

• jobs that re-ran on that day

Dimensions Assigned to

X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Time-Day N/A Total Number of Jobs None - displays all parsed data



Calculation

1. For each day, calculate the number of jobs that re-ran on that day:The job count is calculated differently for jobs that were re-ran (R accounting logrecord). If the start time and end time are different, then the job count will be incre-mented. The below table provides an example of how this calculation is done:

2. For each day, calculate the following:No. of S+E paired records + No. of Only S records + No. of Only E records +No. of D+E paired records + No. of R records

where a S record = job started

E record = job ended

D record = job deleted

R record = job re-ran

Table 1-1: Job count calculation for jobs that re-ran

Acct log date/timestamp

Acct log record type

Job Id

Epoch Start Time

Epoch End Time

Job Count

03/12/2014 14:46:20 Q 1.serverName

03/12/2014 14:46:20 S 1.serverName 1394649980

03/12/2014 14:47:06 R 1.serverName 1394649980 1394650026 1

03/12/2014 14:47:06 S 1.serverName 1394650026

03/12/2014 14:48:06 R 1.serverName 1394650026 1394650086 2

03/12/2014 14:48:07 S 1.serverName 1394650087

03/12/2014 14:49:12 R 1.serverName 1394650087 1394650087 ignored

03/12/2014 14:49:12 S 1.serverName 1394650152

03/12/2014 14:59:22 E 1.serverName 1394650152 1394650762 3



Figure 1-1:Jobs Count by Day



1.2 Jobs by Software by Group

This XYZ chart shows the software usage by custom group.

Calculation

Figure 1-2:Jobs by Software by Group


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Software Group Total Number of Jobs None - displays all parsed data

1. For each custom group, count the total number of jobs for each software(resource_list.software).



1.3 Jobs by Software by Node

This XYZ chart shows the software usage by node.

Calculation

Figure 1-3:Jobs by Software by Node


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Software Node Total Number of Jobs None - displays all parsed data

1. Find all jobs for a software (resource_list.software).2. Group the jobs by node by extracting the node value of each of job from exec_vnode.3. Count the number of jobs for each node.



1.4 Requested Vs Used Memory by Software

This line column chart compares the average requested memory (KB) to the average used memory (KB) for each software.

Calculation

Figure 1-4:Requested Vs Used Memory by Software


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Software N/A Average Memory Used (KB)

Average Memory Requested (KB)

None - displays all parsed data

1. For each software (resource_list.software):

a. Extract the resources_used.mem value for all the finished (E records) jobs.

b. Calculate the average used memory.

c. Extract the Resource_List.mem value for all the finished (E) jobs.

d. Calculate the average requested memory.


2 Cost Analysis

The Cost Analysis category is targeted for the finance department for billing support. The charts showcase resource usage during prime and non-prime hours for each node class.

While analyzing prime/non-prime charts, note that a job may start in one time period and end in another. In this case, CPU/node utilization will be calculated on the basis of resources used by the job during the specific time period. The day is divided into the following three time periods:

• Non Prime Morning: 12:00 AM to 9:00 AM

• Prime Time: 9:00 AM to 5:30 PM

• Non Prime Evening: 5:30 PM to 12:00 AM

For example:

Job 1 walltime will be calculated for Non Prime Morning

Job 2 walltime will be calculated for Non Prime Morning and Prime Time.

Job 3 walltime will be calculated for Prime and Non Prime Evening.

Job 4 walltime will be calculated for Non Prime Morning, Prime, Non Prime Evening.


• Node Walltime by Group - Non Prime Time- walltime by group during non prime time

• Node Walltime by Group - Prime Time- walltime by group during prime time

Table 2-1: Jobs running in multiple time periods

Non Prime Morning

12:00 AM to 9:00 AM

Prime Time9:00 AM to 5:30 PM

Non Prime Evening

5:30 PM to 12:00 AM

Job 1 Starts and ends in Non Prime Morning

Job 2 Starts in Non Prime Morning and ends in Prime Time

Job 3 Starts in Prime Time and ends in Non Prime Time Evening

Job 4 Starts in Non Prime Morning and ends in Non Prime Evening

PBS Analytics 12.3 Chart Calculations10

The node usage percentage is calculated as follows:

Assumption:

Node walltime used for the groups G1, G2, and G3, in the below example, is for illustra-tion purposes only and are not actual numbers. The calculation for node walltime used is described in the calculation example of each chart.

Number of nodes in a cluster = 3 (node1, node2 and node3)

Number of CPUs on each node = 12

Cluster availability = Cluster is available Monday - Friday

= 5 Days (convert to hours = 5 * 24 = 120 hrs)

Node walltime availability = No. nodes * No. CPUs * Cluster availability

= 3 nodes * 12 CPUs * 120 hrs = 4320 hrs

Note: In case the number of groups are more than 150, then as per Pie chart property, the percentage shown on the Pie charts are Pie percentage not the actual utilization. Actual usage in percentage can be viewed by disabling data decimation (refer to section “Config-uring plot point decimation” in the PBS Analytics Administrator’s Guide for more informa-tion on how to change the decimation value).

Table 2-2: Node usage percentage calculation

Group Name Node walltime usedPercentage calculation[node walltime/node availability] * 100

G1 1000 hrs [1000/ 4320] * 100 = 23.14%

G2 120 hrs [120/4320] * 100 = 02.77%

G3 400 hrs [400/4320] * 100 = 09.25%

Unused

[This is not a group but categorized as one]

4320 - 1000 - 120 - 400 = 2800 hrs

[2800/4320] * 100 = 64.81%


2.1 Node Walltime by Group - Non Prime Time

This 3D pie chart displays the node usage percentage by group during non-prime time hours.

Calculation

DimensionsAssigned toGroup By

Values Assigned to

Y AxisChart Filters

Filter Value

Group Total Node Walltime (HRS) Node Event Type Job Busy

Unused

Prime - Non Prime Holiday & Weekend

Non Prime Evening

Non Prime Morning

1. Find all jobs for a specific group.2. Group the jobs by node by extracting the node value for each job from exec_vnode.3. Extract the job start time.4. Segregate the jobs that fall into the non-prime time and holiday window:

a. Convert the date and time to EPOCH date and time format

b. Reference the job start time and end time. If the job started during Prime Time andfinished during Non-Prime Time, then node utilization will be calculated for Non-Prime time only.

c. Calculate the wall time for each job [end - start]*

d. Check the placement of the job:

If placement is exclusive, calculate the node wall time as:

[wall time ] * [total number of CPUS on the nodes on which the job ran]

If placement is free, calculate the node wall time as:

[wall time ] * [total number of CPUs the job consumed]

*typical calculation for walltime - see “Walltime calculations” on page 1 for informationabout walltime calculations for corner cases.


Figure 2-1:Node Walltime by Group - Non Prime Time


2.2 Node Walltime by Group - Prime Time

This 3D pie chart displays the node usage percentage by group during prime time hours.

Calculation

DimensionsAssigned toGroup By

Values Assigned to

Y AxisChart Filters

Filter Value

Group Total Node Walltime (HRS) Node Event Type Job Busy

Unused

Prime - Non Prime Prime

1. Find all jobs for a specific group.2. Group the jobs by node by extracting the node value for each job from exec_vnode.3. Extract the job start time.4. Segregate the jobs that fall into the prime time window:

a. Convert the date and time to EPOCH date and time format

b. Reference the job start time and end time. If the job started during Prime Time andfinished during Non-Prime Time, then node utilization will be calculated for Prime Time only.

c. Calculate the wall time for each job [end - start]*

d. Check the placement of the job:







Figure 2-2:Node Walltime by Group - Prime Time


3 Green Computing

The charts in this category will be used by the finance managers and IT managers to analyze compute node idle time.

This information can be used for cost savings by taking corrective actions such as bringing an unused node down using the PBS Professional Green Provisioning Toolkit.


• Unused Node Capacity by Node- unused walltime by node by time period

• Unused Node Capacity by Quarter - unused walltime by node class by quarter

• Unused Vs Used Node Cycle - used vs unused node walltime by node

Green CPU Walltime calculation example:

Total number of CPUs in a node [ node1 ] = 12

The available node walltime will be calculated based on the following time windows:

Table 3-1: Available node walltime calculation

Time Window Available node walltime

Non Prime Morning(12:00 AM to 9:00 AM) 12 (CPU) * 9 hrs = 108 hrs

Prime Time (9:00 AM to 5:30 PM) 12 (CPU) * 8.5 hrs = 102 hrs

Non Prime Evening(5:30 PM to 12:00 Midnight) 12 (CPU) * 6.5 hrs = 78 hrs


Assume the following jobs ran on node1:

To calculate the Green CPU wall time:

1. Check the start time of these jobs and segregate them into time windows.2. When there are no running jobs in the specific time window, calculate the node wall-

time available on that window as Green CPU walltime.

Based on the information presented in the tables previously, the following conclusions can be made:

Green CPU wall time of 0 indicates that a least a job was present during that time window.

Table 3-2: Jobs running on node1

Job ID

EPOCH Time Human Readable TimeTime

WindowStart time End Time Start time End Time

1 1301805525 1301805600 03 Apr 2011 04:38:45 03 Apr 2011 04:40:00 Non–Prime Morning

2 1301837925 1301838000 03 Apr 2011 13:38:45 03 Apr 2011 13:40:00 Prime

3 1301913525 1365096288 04 Apr 2011 10:38:45 04 Apr 2013 17:24:48 Prime

4 1365157848 1365204288 05 Apr 2013 10:30:48 05 Apr 2013 23:24:48 Prime + Non- Prime Evening

Table 3-3: Green CPU Walltime per time period

DateGreen CPU WalltimeNon–Prime Morning

Green CPU WalltimePrime

Green CPU WalltimeNon–Prime Evening

03 Apr 2011 0 0 78

04 Apr 2011 108 0 78

05 April 2011 108 0 0


3.1 Unused Node Capacity by Node

This stacked column chart displays unused walltime (in days) by node for the time periods:

• Non Prime Morning

• Prime

• Non Prime Evening

• Holiday and Weekend

.Calculation

Dimension

Assigned to

X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Filter Value

Node Prime-Non Prime Total Green CPU Walltime (DAYS) Node Event Type Unused

1. Find all jobs for a specific node by referring to exec_vnode.2. For each quarter, extract the time windows [ prime, nonprime - morning , nonprime-

evening or holiday & weekend ].3. Determine if there were any windows in which there was no running jobs.


Figure 3-1:Unused Node Capacity by Node


3.2 Unused Node Capacity by Quarter

This stacked column chart displays unused node capacity (in days) by quarter.

.Calculation

Figure 3-2:Unused Node Capacity by Quarter

DimensionAssigned

toX Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Filter Value

Time-Quarter Prime-Non Prime Total Green CPU Walltime (DAYS) Node Event Type Unused

1. Find all jobs for a specific node by referring to exec_vnode.2. Group these jobs by yearly quarters (Q1, Q2, Q3, Q4).3. For each quarter, extract the time windows [ prime, nonprime - morning , nonprime-

evening or holiday & weekend ].4. Calculate the Green CPU walltime in each of these time windows (use EPOCH Time).


3.3 Unused Vs Used Node Cycle

This stacked column chart displays unused node walltime (in hours) by node:

.Calculation

DimensionAssigned

toX Axis


Z Axis

Values Assigned to

Y AxisChart Filter

Node Node Event Type Total Node Walltime (HOURS) None - displays all parsed data

1. Calculate the Used Node Walltime by summing the node walltime for all jobs that ranon the node = 120 hrs + 12 hrs + 30 hrs = 162 hrs.

2. Calculate the Unused Node Walltime = Total Node Walltime - Used Node Walltime =1440 - 162 = 1278 hrs.

For this chart assume the following jobs have ran on node1:

Assumptions:

• Total number of CPUs in a node [ node1 ] = 12

• Total number of days the node was available on cluster = 5 days = 5 * 24 hrs = 120 hrs

• Total Node Walltime available on node1 = 120 hrs * 12 CPUs = 1440 hrs

Table 3-4: Jobs running on node1

Job ID CPU used Run Duration (hrs) Node walltime (hrs)

1 12 10 120

2 4 3 12

3 6 5 30


Figure 3-3:Unused Vs Used Node Cycle


4 Productivity

The charts in this category are designed to be used by engineering managers to maximize pro-ductivity and utilization.


• Average Wait Time Vs Average Overall Time by Software - average wait time vs averageoverall time per software

• Average Wait Time by Software by User - average wait time by software/user

• Jobs by Exit Status- total number of jobs by exit status

• Successful Vs Unsuccessful Jobs by Node- node utilization based on successful andunsuccessful jobs

• Successful Vs Unsuccessful Jobs by Software- successful jobs vs unsuccessful jobs bysoftware

• User Job Efficiency and Productivity- walltime per user by exit status


4.1 Average Wait Time Vs Average Overall Time by Software

This line chart compares the average wait time (in seconds) to the average overall time (in seconds) for each software.

.Calculation


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Software N/A Average Wait Time (SECS)

Average Overall Time (SECS)

None - displays all parsed data

1. Find all jobs that are finished (E) for each software using resource_list.software.

a. Extract the wait time as [job start time (stime) - job queued time (qtime)] for eachjob

b. Calculate the average wait time for each software.

2. Find all jobs for each software using resource_list.software

a. Extract the overall time as [job end time - job creation time (ctime)] for each job

b. Calculate the average overall time for each software.


Figure 4-1:Average Wait Time Vs Average Overall Time by Software


4.2 Average Wait Time by Software by User

This XYZ chart displays the average wait time by software for each user.

.Calculation

Figure 4-2:Average Wait Time by Software by User

DimensionAssigned to

X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Filter Value

Software User Average Wait Time (HRS) Node Event Type Job busy

Filter:

Top 10 results on User

1. Find all finished (E) jobs for each software using the value of resource_list.software.2. Group software jobs by user.3. For each software/user grouping:

a. Extract the wait time as [job start time (stime) - job queued time (qtime)] for eachjob.

b. Calculate the average wait time of all jobs within the software/user grouping.


4.3 Jobs by Exit Status

This column chart displays the total number of jobs for each exit status. This chart can be used to highlight the number of jobs that failed or succeeded.

.Calculation

Figure 4-3:Jobs by Exit Status


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Exit Status N/A Total Number of Jobs None - displays all parsed data

1. Find all jobs for each exit status.2. Count the jobs in each exit status category.


4.4 Successful Vs Unsuccessful Jobs by Node

This stacked column chart displays a summary of node utilization based on successful and unsuccessful jobs.

.Calculation

Successful and unsuccessful are configurable parameters and can be con-figured using the pbsa-config-exits command. By default PBSA configures an exit code of “0” as Successful, and non-zero exit codes as Unsuccessful..


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Exit Reason Node Total Number of Jobs None - displays all parsed data

Filter:

Successful

Unsuccessful

1. Find all jobs for each node using the value of exec_vnode.2. Segregate the jobs that have an exit status of "0":

a. Count the number of these job records.

3. Segregate the jobs that have exit status other than “0”:



Figure 4-4:Successful Vs Unsuccessful Jobs by Node


4.5 Successful Vs Unsuccessful Jobs by Software

This stacked column chart compares successful jobs to unsuccessful jobs by software. .

Calculation

Successful and unsuccessful are configurable parameters and can be config-ured using the pbsa-config-exits command. By default PBSA config-ures an exit code of “0” as Successful, and non-zero exit codes as Unsuccessful..


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Exit Reason Software Total Number of Jobs None - displays all parsed data

Filter:

Successful

Unsuccessful

1. Find all jobs for each node using the value of resource_list.software.2. Segregate the jobs that have an exit status of "0".


3. Segregate the jobs that have exit status other than “0”.



Figure 4-5:Successful Vs Unsuccessful Jobs by Software


4.6 User Job Efficiency and Productivity

This stacked column chart compares job failure rates to successful jobs for each user in rela-tion to usage (walltime).

.Calculation


X Axis


Z Axis

Values Assigned to

Y AxisChart Filters

Exit Reason User Total Node Walltime (HRS) None - displays all parsed data

Filter:

Top 20 results on User

1. Find all jobs for a specific user2. Group these jobs by node using the value exec_vnode3. For each job within the user/node grouping:

a. Group the jobs that have exit codes configured as Still Running, Successful, andUnsuccessful.

b. For each grouping:

i. Calculate the wall time for each of these jobs [end - start]*

ii. Check the placement of the job:







Figure 4-6:User Job Efficiency and Productivity


pbs analytics 12.2 chart calculations...

Documents