kpi project final report - intosai working group on...
TRANSCRIPT
![Page 1: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/1.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
KPI Project Final Report — Key Performance Indicators Methodologyfor Auditing IT Programs
Amy Young, IT Center
China National Audit OfficeApril, 2013 Lithuania
![Page 2: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/2.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 2
CNAOFramework
1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
![Page 3: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/3.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 3
CNAO1. Background
•the 19th of the INTOSAI Working Group on IT Audit , April 2010
– three proposed projects:
• Development of IT Performance Indicators
• Performance measures of IT Solutions implemented in government organizations
• Index System about IT Performance Audit
![Page 4: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/4.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 4
CNAO1. Background
• the 19th of the INTOSAI Working Group on IT Audit , April 2010
– The above three projects were put into a new one. “Key Performance Indicator Methodology for Auditing IT Programs”.
– SAI China volunteered to be the team leader.
– Team members : SAI Bhutan, China, Ecuador, Japan, Kuwait, Malaysia, Pakistan, Poland, Russia and USA.
![Page 5: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/5.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 5
CNAOFramework
1. Background
2. Project objectives
3. Project progress
4. Project products
![Page 6: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/6.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 6
CNAO2. Project objectives
• two principal products
– a set of specific and measurable IT-related indicators
– guideline
•the ideas of KPI application
•feasible KPI evaluation methodology
![Page 7: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/7.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 7
CNAO2. Project objectives
• Analysis, conclusions and decision making based on results of performance measurement
•exchange views, know-how, and information
![Page 8: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/8.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 8
CNAOFramework
• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
![Page 9: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/9.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 9
CNAO3. Project progress
• Milestones
– By the end of 2011, the beta database of KPI indicators.
– By the end of 2012, the draft Guideline.
– In 2013, the final Guideline and KPI Database.
![Page 10: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/10.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 10
CNAO3. Project progress
• the first team meeting on Apr 16, 2010 in Beijing.
• a new name for the project, “Key Performance Indicators Methodology for Auditing IT Programs”.
• the minutes of the first team meeting.
![Page 11: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/11.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 11
CNAO3. Project progress
• SAI Russia, Japan , USA , Kuwait and Ecuador explained their own understanding about the project from April to June, 2010.
• The project initiation document to Chair India in July 2010.
• SAI Kuwait submitted the beta database of KPI indicators on Dec 29, 2010.
![Page 12: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/12.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 12
CNAO3. Project progress
• The original framework based on SAI Kuwait’s and China’s PKI indicators in Feb, 2011.
• SAI Kuwait, Bhutan and Japan gave the comments in Feb and Mar, 2011.
• the Second team meeting on Apr 16, 2011 in South Africa.
• the minutes of the second team meeting.
![Page 13: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/13.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 13
CNAO3. Project progress
• SAI Kuwait submitted the categorized KPI beta database on August 11, 2011.
• SAI Bhutan gave comments on August 18 ,2011.
• SAI China composed the beta indicators and send it to the team members for comments on November 30, 2011.
• SAI Japan and Kuwait gave the comments in
December,2011.
![Page 14: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/14.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 14
CNAO3. Project progress
• In the 21st meeting in Kuala Lumpur, Malaysia in January 2012, the team made the report about the KPI project.
• SAI China sent the content framework of guideline to the team members for comments on April 9, 2012.
• In April and May, SAI Kuwait and Bhutan gave the comments and SAI Kuwait applied to compose Part 1 and Part 2.
• SAI China began to compose the draft guideline in June, 2012.
![Page 15: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/15.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 15
CNAO3. Project progress
• SAI China delivered the draft guideline for comments among the team members on December 30, 2012.
• SAI Japan, Kuwait and USA feedback in Jan and Feb 2013.
• In early March 2013, SAI China send the final guideline v1 to Chair India for comments among all the WGITA members.
![Page 16: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/16.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 16
CNAO3. Project progress
• SAI Bangladesh, Brazil, Iraq and Russia gave the comments.
• Based on the comments, SAI China made the final guideline v1.1.
• Comments from SAI Malaysia, Lithuanian and Qatar.
• In Apr 2013, the final guideline V1.2 was delivered for approval in the 22nd meeting of WGITA .
![Page 17: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/17.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 17
CNAO3. Project progress
• Communication Management Strategy
– mainly based on email through Internet
– email contact list for the team members
– sometimes discuss trough telephone
![Page 18: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/18.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 18
CNAOFramework
• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
![Page 19: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/19.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 19
CNAO4. Project products
• Criteria for the indicator
– The indicators: be widely accepted and measurable
– the data source of the evaluating indicators : be accessible and analyzable
– the calculation methods of the evaluating indicators: be relatively simple
![Page 20: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/20.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 20
CNAO4.1- KPI database
• 11 parts, 367 indicators with three-level.
• 1. Decision (15 indicators)
– Compliance with the laws
– Feasibility study
– Participation in decision
• 2. Requirement analysis (13 indicators)
– Organization target
– Core business coverage
– Response/change
![Page 21: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/21.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 21
CNAO4.1- KPI database
• 3. Design / Planning (45 indicators)
– Requirement coverage
– Time limit
– Capacity Planning and Resource Provisioning
– Cost estimation
– IT risks
![Page 22: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/22.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 22
CNAO4.1- KPI database
• 4. Procurement/Development (48 indicators)– Selection for partner or supplier
– Cost control
– Process Control
– Code control
– Outsourcing
– Quality Control
– Testing
– Training
– Upgrading
![Page 23: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/23.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 23
CNAO4.1- KPI database
• 5. Product (26 indicators)
– User satisfaction
– Price
– Delivery
– Performance
– Integration
– Technology applicability
![Page 24: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/24.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 24
CNAO4.1- KPI database
• 6. Maintenance (125 indicators)
– Follow the management rules
– Incident management
– System Usability
– Availability
– Maintenance cost
– Website
– Monitoring
– Change Management
– Data Center
![Page 25: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/25.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 25
CNAO4.1- KPI database
• 7. Security (26 indicators)
– Security plan
– Identity management
– User account management
– Security testing monitoring
– Security incident definition
– Malicious software prevention
– Network security
![Page 26: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/26.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 26
CNAO4.1- KPI database
• 8. Backup and Disaster Recovery (14 indicators)
– Backup plan
– Recovery plan
– Backup operation management
– Recovery operation management
• 9. Service (32 indicators)
– Service request
– Service Response
– Service Satisfaction
![Page 27: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/27.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 27
CNAO4.1- KPI database
• 10. Effectiveness (19 indicators)
– Coverage of the core business
– Benefit
– Internal Management optimizing
– Public Service
• 11. Others (4 indicators)
![Page 28: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/28.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 28
CNAO4.2 - Guideline
• Part 1. Preface
• Part 2. Audit Plan
• Part 3. Audit Implementation
• Part 4. Audit report
• Reference : INTOSAI training materials, ISACA,
COBIT, ITIL and others.
![Page 29: KPI Project Final Report - INTOSAI Working Group on …intosaiitaudit.org/WGITA22nd/22thWGITAMeeting/Presentations/2. KPI...the 22nd meeting of the INTOSAI Working Group on IT Audit](https://reader031.vdocuments.mx/reader031/viewer/2022022010/5afccd427f8b9a434e8ca964/html5/thumbnails/29.jpg)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
Thanks
for your attention!
China National Audit OfficeApril, 2013 Lithuania