day 1 1620 - 1705 - pearl 1 - deep kamal singh

1

Strategic approach to ascertain accurate decisions after unplanned service outage in Telecom operations

Deep Kamal Singh

27 September 2013

2

Introduction

• Changes in System are Inevitable

• Uncertainty, Variations, Deviations are Un Avoidable

• Prime objective of Business – Profit & Cost Control

• Factors that control cost of variation/deviation – Number of interdependent systems/services disrupted– Knowledge Gap between cross functional domains– Time spent in reaching a decision

• Exponentially proportional to Stake holders involved• Directly proportional to number of services/systems

disrupted

• Need of the hour, Business Need


3

• Organizations are facing a complex and

risky environment–A “do more with less”– “Tight” economic climate–Expanding global priorities–Necessity to enable innovation

The Strategic Approach

Identify All Services

Identify Revenue of

every service

Define Threshold

Form Crisis Management

Team


Identify All Impacted Services

Estimate outage & Calculate Revenue

Consolidate Options

Decide• Delay• GO Live• Fall Back

Planning

Execution - Handling Crisis

Organizations are facing a

complex and risky environment– A “do more with less”– “Tight” economic climate– Expanding global priorities– Necessity to enable

innovation

4Presentation Title

Knowledge Area Processes

Initiating Process Group

Planning Process Group Executing Process GroupMonitoring

&Controlling Process Group

Closing Process Group

Project Integration Management

4.1 Develop Project Charter

4.2 Develop Project Management Plan

4.3 Direct and Manage Project Execution

4.4 Monitor and Control Project Work 4.6 Close

Project or Phase 4.5 Perform Integrated

Change Control

Project Scope Management

5.1 Plan Scope Management 5.5 Validate Scope 5.2 Collect Requirements 5.6 Control Scope 5.3 Define Scope 5.4 Create WBS

Project Time Management

6.1 Plan Schedule Management

6.7 Control Schedule

6.2 Define Activities 6.3 Sequence Activities 6.4 Estimate Activity Resources 6.5 Estimate Activity Durations 6.6 Develop Schedule

Project Cost Management

7.1 Plan Cost Management 7.4 Control Costs

7.2 Estimate Costs7.3 Determine Budget

Project Quality Management

8.1 Plan Quality Management 8.2 Perform Quality Assurance

8.3 Perform Quality Control

Project Human Resource Management

9.1 Plan Human Resource Management

9.2 Acquire Project Team

9.3 Develop Project Team

9.4 Manage Project Team

Project Communications Management

10.1 Plan Communications Management

10.2 Manage Communications10.3 Control Communications

Project Risk Management

11.1 Plan Risk Management

11.6 Monitor and Control Risks

11.2 Identify Risks 11.3 Perform Qualitative Risk Analysis11.4 Perform Quantitative Risk Analysis11.5 Plan Risk Responses

Project Procurement Management

12.1 Plan Procurement Management

12.2 Conduct Procurements 12.3 Control Procurements

12.4 Close Procurements

Project Stakeholder Management

13.1 Identify Stakeholders

13.2 Plan Stakeholder Management

13.3 Manage Stakeholder Engagement

13.4 Control Stakeholder Engagement

PMI PMBOK™ V.5

5

Agenda

Introduction

Activities and Phases

Being prepared for uncertainty

Handling uncertainty

Theory in practice

In Conclusion

Q & A


6


• Complexity Assessment is first step to prepare for

uncertainty

• Concise definition of Decision makers for every stages

• Knowing Phases of activity – Strategic approach to take

next step under unplanned outage/failure depends upon the

phase– 3 phases of any activity in simplistic view

• Preparation • Execution• Operation


7

Activity Complexity

1 2 3 4 5 6 7

Complexity 1 4 9 16 25 36 49

Teams 1 2 3 4 5 6 7

5

15

25

35

45

55

Activity Complexity Vs. Systems Involved

Co

mp

lex

ity


• Complexity of activity– Exponentially proportional to

number of Inter-dependent systems involved

– Directly proportional to number of cross functional teams involved

– Directly proportional to Allocated duration versus tasks list ratio

8

Impact of failure during Activity phases

Presentation Title

Planning Phase

• Delayed Activity timelines

• Budget overhead

• Resources re-alignment

• Impact on cost to business

• Man hours invested in

preparation, development,

planning, testing etc.

• Business loss due to delayed

launch

• Delay overhead cost on

logistics

• Cascading delay on other

planned activities

Execution Phase

• Increased service downtime

Sudden revenue loss


• Direct revenue loss

• Customer dissatisfaction

Unpredictable future damages

• Increased calls to customer

care and support systems

• Resource working in

extended hours

Unpredictable future issues.

• If Fallback is decided Total

investment sunk.

Operation Phase

• Live Service downtime

Sudden revenue loss


• Direct revenue loss

• Customer dissatisfaction

Unpredictable future damages

• Increased calls to customer

care and support systems

• Unplanned additional Man

hour investment

• Resource working in

extended hours

Unpredictable future issues.

9

Agenda

Introduction




Theory in practice

In Conclusion


10

Activity Planning

• The present day Approach– Preparation, Review and approval of detailed step wise

execution plan• Estimation of planned outage duration.

– Business teams analyzes and calculate total cost of activity to business • Including cost of planned outages, consequential costs,

logistics, HR overhead and so on.– Every step of activity is executed, monitored and controlled if

everything goes ‘as per plan’

• Shortcomings – We can not predict what may go wrong and thus

we can not be 100% ready for unplanned outages


11

Planning for uncertainty

Being prepared for something that can not be planned – During

Preparation Phase

• Identification of all system and services

• Evaluation of revenue earnings of each services

• Formation of Crisis management team

• Unanimous view of Project managers, Activity owners, Team

leads and Business teams over – Revenue potential of each service– All factors that impact business for each service

• Documentation of the unanimous view – “ Service Impact

Matrix”


12

Preparing Service impact matrix

• “Service impact matrix” - single document of reference to assess

impact to business during outage

– List all services which are going to be affected due to the

planned activity

– Include subscriber/user/consumer base of every service

– Include time based distribution of revenue earnings of each

service,

– Include direct and indirect cost to business for each service if

failure is encountered at any time duration

• Service Impact Matrix must be finalized, reviewed and approved by

all stake holders.

– Saves time to estimate cost of outage to business during crisis

– Prevents lengthy review/approval process


13

Example Service impact matrix

SNO Service NameSubscriber

Base(In Mn)

Hourly Revenue Potential [in 100K Rs.] Indirect cost to business

H01 H02 H03 H04 H… H21 H22 H23 H24Customer Care Overhead

Dependent Services impact

Customer Satisfaction Impact

1 Voice Calls 14.34 …

1.1 Local Home network 14.34 …

1.2 Local cross network 14.34 …

1.3 National Long Distance 09.60 …

1.4 International Long Distance 00.12 …

2 SMS 14.34 …

2.1 Local 02.21 …

2.2 National 02.21 …

2.3 International 00.01 …

3 Data Usage 01.90 … 4 Paper Recharges 08.14 … 5 ETOPUP 11.89 … 6 USSD Services 14.34 …

6.1 Subscriber Info 14.34 …

6.2 Subscription management 04.30 …

6.3 VAS Services 01.80 …

7 Subscriber Life Cycle 14.34 …

7.1 Subscriber Activation 0.03 …

7.2 Subscriber Churn 00.01 …

7.3 Service management 00.01 …


14

Agenda

Introduction




Theory in practice

In Conclusion

Q & A


15

Identifying affected services and subscriber base

In case of any unexpected outage we need to find out total impact

to business, thus

• First and foremost, get update from all teams and consolidate

list of services that are affected.

• Classify level of disruption for each impacted service– Intermittent, Partial or Total

• Estimate outage duration and dependency factors for the

impacted service

• Consolidated outage stats for each impacted service in a single

view for comparison and correlation.


Considering the revenue impacting factors associated with the activity, below equation can be deduced

If

R(S1), R(S2)…..R(Sn) = Avg. revenue per user per minute of Service1,2,…n

O(S1),O(S2)…..O(Sn) = Total estimated outage duration of Service1,2,…n

Sb(S1),Sb(S2)….Sb(Sn) = Estimated percentage of affected subscriber base during outage of Service1, 2,….n

Su(S1), Su(S2)…..Su(Sn) = Total users of Service1, 2, ….n

Then

Total loss to business during service outage =

∑ [ R(Sn) x O(Sn) x Sb(Sn) x Su(Sn) / 100 ]

16

Deciding next course of action

• Identify impacted services

• Calculate Revenue Impact– Consolidate outage stats of all affected

services– Correlate available data with corresponding

details in Service Impact Matrix

• Assess Variable and Dependency factors

• Mathematically deduce all possible next steps

to formulate “The Decision Matrix”


17

Formulating Decision matrix

• The Decision matrix – Mapping of Options vs. Risk vs. Revenue Impact vs.

Variance factor – Crisis Management team to prepare

• Cross functional managers and tech leads to tabulate

domain specific next step of actions

• Categorization of options– Track 1 :: Go-Live– Track 2 :: Delay– Track 3 :: Fall back


18

Finalizing Decisions

• Ignore – impossible, unattainable, high risk options– Rule out options which are

• least possible, • have highest risk, • have highest number of variable factors

• Final time estimation– Get ‘Time to implement’ from respective teams– Readiness Confirmation – technically and logistically

• Final out put of crisis management team:-

Decision matrix with final concluded set of options and details of

‘time to implement’ presented to apex decision makers


19

Agenda

Introduction




Theory in practice

In Conclusion

Q & A


20

Case study : Activity Details

• Billing System upgrade in 14 telecom circles of a leading telecom operator in India

• The activity requires complete outage of billing system for 8 hours, Since billing

system stays unavailable, all connected services (as listed) also face subsequent

downtime:– Voice calls – Local / National Long distance / International long distance– SMS– Data Browsing– Real-time data charging– Recharges – Voucher recharge and E-topup– USSD– Unified Subscriber life cycle management – Activation, Churn, daily jobs, and other offline process.– Business reporting – MIS

• The complex activity involves changes at many functional systems – High level of coordination and control required between Cross functional

technical teams.– Due to numerous Inter-dependent systems, outage of one system or service

may result in consequential outage of other services also


21

Case study 1

• Subscriber : 5.4

• Activity: Core billing system upgrade

• Issue during execution: ETOPUP not working with new version– Important to note here that multiple rounds of testing on test-bed and rehearsals were

conducted which showed no sign of issues earlier.

• Technical teams started investigating and fix the issue ASAP.

• Business team kept waiting for updates from Technical team to decide next step.

• All possible next actions were greatly debated without reaching a final conclusion– ETOPUP : DELAY

• Given estimated time to fix the issue was 2 Hours– Revenue assurance team : FALL BACK

• Cancelling complete activity and reinstating old version billing system– Billing system team : GO LIVE

• leaving ETOPUP team to fix “their” issue separately

• Business team: absence of conclusive data No decision was taken

• Finally after 95 minutes of extended downtime, ETOPUP issue was resolved and then upgraded

version of billing system was made live.


22

Evolution of strategic Approach

• Learning from failure – What took time to reach a decision?– Lack of information to take a decision

• “What to do next” is Biggest Time eater– Lack of predefined delay threshold for critical milestones– Lack of designated managers to manage crisis

• Biased Decision Making is biggest Hindrance to consensus• Absence of a dedicated Crisis management team.

– Lack of clarity on activity owners, team leaders, business owners and decision makers• Dedicated team lead/managers always suggest biased options which

favor their own team– Lack of methodical approach to ensure quick decision making

The learning drawn from events of site 1 became foundation for evolution of

strategic approach which is discussed in this presentation.


23

Case study 2

• Subscriber Base : 9.28 million

• Activity Details: Same activity as Case study 1

• The Issue: Once new version of billing system was made live for test traffic– It was identified that some of call handling units were not functioning

• When call routed on those units voice calls, video calls, SMS, USSD were failing

– Other units were functioning without any issues

• Crisis management team swung in action and as a first step it was deduced

that 20% of units were not functioning– This meant Voice calls, SMS and USSD requests were impacted by 20%.

Important to note that the Crisis management team was doing all coordination

and fact gathering with cross functional technical and business teams, the

Technical teams were dedicatedly focused to resolve the issue.


Continued…

24

Case study 2

• Total Impact identified Service impact matrix referred Assessment– Revenue loss when Billing system is made live with 20% failures in 3 service– Cost of delaying the launch – Which meant 100% subscriber base will not be able to

use ANY service until the issue is fixed– Cost of fallback was also compared with above two costs

• 3 Tracks Options (Delay, Go Live, Fallback ) consolidated – Presented to Business owners and Function heads

• Decision Taken– Within the duration of 45 minutes allocated for system testing before launch –

Business team successfully reached a decision to remove those 20% units from network (so call will not land on them) and make the system Live as per schedule

Even with unexpected variations encountered during critical launch

schedule, the strategic approach helped reaching a right decision

methodically


25

Agenda

Introduction




Theory in practice

In Conclusion

Q & A


26

Identify impacted services

Identify impacted subscriber/user base• Estimate

probable service outage duration for each impacted service

Evaluate indirect, additional and future cost to

business due to service outages

Formulate list of possible next steps

or possible options.

Identify variable (uncertain) factors

and Risk associated with each concluded

option.

Prepare Option vs. Risk vs. Revenue Impact vs. Variance factor matrix having each of 3 tracks• Track 1 - Go-

Live• Track 2 – Delay• Track 3 – Fall

backEvaluate time to

implement for every option.

Present Decision makers with Tracks

containing concluded set of

options and details of ‘time to

implement’ for each option.

Conclusion

27

Thank You

Deep Kamal Singh

Project Manager – BSS APAC

Comverse Network Sys.

[email protected]

+91-9971003585

Shay Shargal, PMP

CEO, RBS Projects

[email protected]

+972-52-23-11-11-0

www.rbsprojects.co.il

Past President – PMI Israel Chapter

http://www.rbsprojects.co.il/