final presentation master’s thesis: tool support for capability … · final presentation...

Software Engineering for Business Information Systems (sebis) Department of InformaticsTechnische Universität München, Germany

wwwmatthes.in.tum.de

Final Presentation Master’s Thesis:Tool Support for Capability-Based Application Portfolio Management- Conceptualization, Prototype Implementation, and EvaluationNovember 28th, 2016Fatih YilmazSupervisor: Pouya Aleatrati Khosroshahi, M.Sc.

Prof. Dr. Florian Matthes

Agenda

2

1

RESEARCH QUESTIONS &APPROACH

2

RELATED WORK

3

APPLICATIONLANDSCAPEEVALUATION

4

VISUALIZATION &PROTOTYPING

5

RESULTS

Final Presentation Master’s Thesis – Fatih Yilmaz

Motivation

3Final Presentation Master’s Thesis – Fatih Yilmaz

We had perhaps more than 80 [KPIs] throughout the business. Our objective was to achieve a increase in KPI visibility across our group, and understand what drives us.

- Statement of a German insurance company [Georges 2013]

You can’t manage what you can’t measure.- Peter Drucker (1909 - 2005)

”What gets measured gets done.- William Thomson (1824 - 1907)

Ever since the beginning of computing there has also been an interest in performance measurement.

- Wiggers (2004)

Measurement important for productivity

Performance measurement crucial in IT

Missing holistic view due to many Key Performance Indicators (KPIs)

”” ”

” ”

””

Research Questions

4

?

?

?

What kind of application attributes can be used to evaluatethe status of application landscapes (AL)?

What kind of operational actions can be derived from the application landscape’s status?

How can the business capability map be used to visualize the application landscape’s status?


Agenda

5

1


2

RELATED WORK

3


4


5

RESULTS


Related Work

6

AL evaluation Aggregated status view

Capability-basedAPM

Aleatrati Khosroshahi (2016) (✓) complexity ✕ ✕

Schneider (2016) ✓ ✕ ✕

Schneider (2015) (✓) complexity ✕ ✕

Podgórski (2014) ✕ ✓ ✕

Schütz (2013) (✓) complexity ✕ ✕

Lagerström (2013) ✓ ✕ ✕

Kandjani (2012) (✓) complexity ✕ ✕

Freitag (2011) ✕ ✕ ✓

Ulrich (2011) ✕ ✕ ✓

Mocker (2009) (✓) complexity ✕ ✕

Saqib (2008) ✕ ✓ ✕

Jollands (2003) ✕ ✓ ✕

Weill (1999) ✓ ✓ ✕


Lack of research onholistic status evaluation of ALs by means of business capabilities

Agenda

7

1


2

RELATED WORK

3


4


5

RESULTS


KPI Aggregation Framework

8

“Subindices” Stated relevance

Number of interfaces [Mo09] [La08] [Schn15] [Schm13] [Re15] [Ki03] [Al16]

Operational costs [me12] [La08] [Industry partner]

Number of incidents [me12] [cap14] [La08]

Capability coverage [Mor02] [Mo09] [Schm13] [Re15]

Strategic relevance [Industry partner]

Application failure [Mor02] [cap14] [Va05]

Incident processing time [Industry partner]

Business impact [Industry partner]

Application age [Mo09] [Be13] [Ro06] [Sc04] [Al16]

Number of users [La08] [Al16]

Number of technological components

[Mo09] [Schn15] [Schm13] [Re15]

Deviation from standard [Mo09] [Bo09] [Schn15]

Weighting schemes:• Direct

monetization• Expert assessment• Public opinion

polls• Distance to target• Distance to policy

target• Cost of distance to

target• Implicit weighting• Statistical

methods

Select weighting scheme

Calculate weights

Calculate subindices

Select subindicesfor inclusion in

aggregation function

Select appropriate aggregation

function

Weights needed

?

Calculate aggregation

function

Report aggregate indices

No

Yes


Based on [Jo03]

9

“Subindices” Stated relevance

Number of interfaces [Mo09] [La08] [Schn15] [Schm13] [Re15] [Ki03] [Al16]

Operational costs [me12] [La08] [Industry partner]

Number of incidents [me12] [cap14] [La08]

Capability coverage [Mor02] [Mo09] [Schm13] [Re15]

Strategic relevance [Industry partner]

Application failure [Mor02] [cap14] [Va05]

Incident processing time [Industry partner]

Business impact [Industry partner]

Application age [Mo09] [Be13] [Ro06] [Sc04] [Al16]

Number of users [La08] [Al16]

Number of technological components [Mo09] [Schn15] [Schm13] [Re15]

Deviation from standard [Mo09] [Bo09] [Schn15]

COMPLEXITY QUALITY IMPACT

Initial objective:One distinct KPI representing the general AL status of the business capabilitiesProblem:Ambiguous statement of the KPISolution:Multiple KPI categories






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

10

Requirements [Industrial partner]

- TraceabilityTraceable function

- RobustnessAdaptable to different use cases

- ComparabilityDifferent market KPIs are comparable






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

11

𝑪𝒐𝒎𝒑𝒍𝒆𝒙𝒊𝒕𝒚+,- =1

𝐴+,-1 (

𝑖_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑖𝑛; + 𝑖_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑜𝑢𝑡; ∗ 𝑃 + 𝑒_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑖𝑛; + 𝑒_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑜𝑢𝑡;⋃ 𝑡𝑜𝑡𝑎𝑙_𝑖𝑛𝑡𝑓𝑎𝑐𝑒𝑠_𝐼𝑁- + ⋃ 𝑡𝑜𝑡𝑎𝑙_𝑖𝑛𝑡𝑓𝑎𝑐𝑒𝑠_𝑂𝑈𝑇-�

-∈K�-∈K

�

;∈MN,O

+

PQ-RST_UV_+UWSTSX_+;Y;RZ[Z\ZS]^⋃ \U\;[_PQ-RST_UV_+;Y;RZ[\ZS]O�O∈_

+;`S^

abc(⋃ {;`S^,O})O∈_

+

PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]^⋃ \U\;[_PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]�O∈_ O

+

(1 − PQ-RST_UV_]\;PX;TX_+U-Y[Z;P\_\S+gPU[U`h_+U-YUPSP\]^PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]^

))

A denotes the set of all applicationsa denotes the observed applicationc denotes the observed business capabilitiesm denotes the observed marketsAc,mdenotes the set of all applications used in the capability c of the market mP denotes the „Penalty“ value for capability internal interfaces (e.g. 0,5)

𝐴+,-:= 𝑎|𝑎 ∈ 𝐴 ∧ 𝑐 ∈ 𝐶 ∧ 𝑚 ∈ 𝑀 ∧ 𝑢𝑠𝑒𝑠(𝑐, 𝑎) ∧ 𝑙𝑜𝑐𝑎𝑡𝑒𝑑(𝑐,𝑚)𝑢𝑠𝑒𝑠 ⊆ 𝐶×𝐴

𝑙𝑜𝑐𝑎𝑡𝑒𝑑 ⊆ 𝐶×𝑀






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

12

𝑸𝒖𝒂𝒍𝒊𝒕𝒚+,-,\ =1

𝐴+,-1 �

;∈MN,O

(1 𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑖𝑛𝑐𝑖𝑑𝑒𝑛𝑡𝑠;,},\ ∗ 𝑎𝑣𝑔_𝑝𝑟𝑜𝑐𝑒𝑠𝑠𝑖𝑛𝑔_𝑡𝑖𝑚𝑒;,},\ ∗ 𝑃)�

}∈�

+𝑠𝑢𝑚_𝑜𝑓_𝑑𝑜𝑤𝑛𝑡𝑖𝑚𝑒𝑠;,\)

𝑰𝒎𝒑𝒂𝒄𝒕+,-,\ =1

𝐴+,-1 (�

;∈MN,O

𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑐𝑜𝑠𝑡𝑠;,\max(⋃ {𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑐𝑜𝑠𝑡𝑠;,-,\})

-∈K +

𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑢𝑠𝑒𝑟;,\

𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑒𝑚𝑝𝑙𝑜𝑦𝑒𝑒𝑠-,\+

𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙_𝑏𝑢𝑠𝑖𝑛𝑒𝑠𝑠_𝑖𝑚𝑝𝑎𝑐𝑡;)+𝑠𝑡𝑟𝑎𝑡𝑒𝑔𝑖𝑐_𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒+

𝑃 =

0,1𝑖𝑓𝑘𝜖𝐼𝑛𝑐𝑖𝑑𝑒𝑛𝑡[U�0,2𝑖𝑓𝑘𝜖𝐼𝑛𝑐𝑖𝑑𝑒𝑛𝑡-SXZQ-0,5𝑖𝑓𝑘𝜖𝐼𝑛𝑐𝑖𝑑𝑒𝑛𝑡gZ`g1𝑖𝑓𝑘𝜖𝐼𝑛𝑐𝑖𝑑𝑒𝑛𝑡+TZ\Z+;[

Exemplary values. Can be adjusted by the expert.

t denotes observed timer periodk denotes incident categoryP denotes the ”penalty” value

𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙_𝑏𝑢𝑠𝑖𝑛𝑒𝑠𝑠_𝑖𝑚𝑝𝑎𝑐𝑡;:= 1,0𝑠𝑡𝑟𝑎𝑡𝑒𝑔𝑖𝑐_𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒+:= 1,0






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

13

“Subindices” Weight

Number of interfaces 0,47

Capability coverage 0,3

Number of technological components

0,1

Deviation from standard 0,1

Application age 0,03


Application failure 0,53

Number of incidents 0,33

Incident processing time 0,13


Operational costs 0,33

Strategic relevance 0,27

Business impact 0,3

Number of users 0,1

COMPLEXITY

QUALITY

IMPACT






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

𝑰𝒎𝒑𝒂𝒄𝒕+,-,\ =1

𝐴+,-1 (�

;∈MN,O

𝑔 𝑖 ∗𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑐𝑜𝑠𝑡𝑠;,\

max(⋃ {𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔_𝑐𝑜𝑠𝑡𝑠;,-,\})-∈K

+

𝑔 𝑢 ∗𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑢𝑠𝑒𝑟;,\

𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑒𝑚𝑝𝑙𝑜𝑦𝑒𝑒𝑠-,\+

𝑔 𝑏𝑖 ∗ 𝑏𝑢𝑠𝑖𝑛𝑒𝑠𝑠_𝑖𝑚𝑝𝑎𝑐𝑡;)+𝑔 𝑠 ∗ 𝑠𝑡𝑟𝑎𝑡𝑒𝑔𝑖𝑐_𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑐𝑒+

14

𝑪𝒐𝒎𝒑𝒍𝒆𝒙𝒊𝒕𝒚+,- =1

𝐴+,-1 (𝑔 𝑖𝑛𝑡 ∗

𝑖_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑖𝑛; + 𝑖_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑜𝑢𝑡; ∗ 𝑃 + 𝑒_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑖𝑛; + 𝑒_𝑖𝑛𝑡𝑓𝑎𝑐𝑒_𝑜𝑢𝑡;⋃ 𝑡𝑜𝑡𝑎𝑙_𝑖𝑛𝑡𝑓𝑎𝑐𝑒𝑠_𝐼𝑁- + ⋃ 𝑡𝑜𝑡𝑎𝑙_𝑖𝑛𝑡𝑓𝑎𝑐𝑒𝑠_𝑂𝑈𝑇-�

-∈K�-∈K

�

;∈MN,O

+

𝑔 𝑐𝑜𝑣 ∗ PQ-RST_UV_+UWSTSX_+;Y;RZ[Z\ZS]^⋃ \U\;[_PQ-RST_UV_+;Y;RZ[\ZS]O�O∈_

+

𝑔 𝑎𝑔𝑒 ∗ ;`S^abc(⋃ {;`S^,O})

O∈_+

𝑔 𝑡𝑒𝑐 ∗ PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]^⋃ \U\;[_PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]�O∈_ O

+

𝑔 𝑑𝑒𝑣 ∗ (1 − PQ-RST_UV_]\;PX;TX_+U-Y[Z;P\_\S+gPU[U`h_+U-YUPSP\]^PQ-RST_UV_\S+gPU[U`h_+U-YUPSP\]^

)

𝑸𝒖𝒂𝒍𝒊𝒕𝒚+,-,\ =1

𝐴+,-1 �

;∈MN,O

( 𝑔 𝑖𝑛𝑐 + 𝑔 𝑡 ∗ (1 𝑛𝑢𝑚𝑏𝑒𝑟_𝑜𝑓_𝑖𝑛𝑐𝑖𝑑𝑒𝑛𝑡𝑠;,},\ ∗ 𝑎𝑣𝑔_𝑝𝑟𝑜𝑐𝑒𝑠𝑠𝑖𝑛𝑔_𝑡𝑖𝑚𝑒;,},\ ∗ 𝑃)�

}∈�

+𝑔 𝑑 ∗ 𝑠𝑢𝑚_𝑜𝑓_𝑑𝑜𝑤𝑛𝑡𝑖𝑚𝑒𝑠;,\)

1𝑔(𝑥)�

�

= 1

1𝑔(𝑥)�

�

= 1

1𝑔(𝑥)�

�

= 1






methods


Calculate weights





function

Weights needed

?


function


No

Yes



Based on [Jo03]

Agenda

15

1


2

RELATED WORK

3


4


5

RESULTS


Mock-Up

5.5.2014Discussion of Deliverable for Milestone 1 16

CAT

1 Cap 1

Cap 1.1 Cap 1.2

Cap 2

Cap 2.1 Cap 2.2

Cap 2.3

Cap 3

Cap 3.1 Cap 3.2

CAT

2

BUSINESS CAPABILITY MAP – APPLICATION COMPLEXITY STATUS 2016Germany

Cap 2.1

Cap 2.3

Cap 2.2

Cap 2.1

5.5.2014Discussion of Deliverable for Milestone 1 17

LIVE DEMO

Agenda

18

1


2

RELATED WORK

3


4


5

RESULTS


Results


Future workKPI Aggregation Visualization

Robustness:• Use a larger data set for reliable multicollinearity

and outliers tests

Flexibility:• Adaptable layout of the business capability map• Colour scale adjustment in the front end

Maturity:• Widen expert evaluation for more precise weighting

Automatization:• EAM tool connection for automatic data transfer

Main contributionKPI Aggregation Visualization

Identification, filtering and categorization of application status evaluating characteristics Continuous requirements elicitation

Development and continuous improvement ofaggregated KPIs Conceptual design of the prototype

Expert evaluation

THANK YOU!ANY QUESTIONS?

Fatih Yilmaz (B.Sc.)

[email protected]

Technische Universität München Department of InformaticsChair of Software Engineering forBusiness Information Systems

Boltzmannstraße 385748 Garching bei München

References

21

[Al16]Aleatrati Khosroshahi, P., Beese, J., & Aier, S. (2016). What drives application portfolio complexity? An empirical analysis of application portfolio cost drivers at a global automotive company. In 18th IEEE Conference on Business Informatics (CBI 2016), Paris.[Be13]Beetz, “Wirkung von IT-Governance auf IT-Komplexitat in Unternehmen, Beeinflussung der IT-Redundanz durch Verantwortungsteilung im T-Projektportfolio,” Wiesbaden, Germany: Springer Fachmedien, 2013.[Bo06]E. Bouwers, J. Visser and A. van Deursen, "Criteria for the evaluation of implemented architectures," Software Maintenance, 2009. ICSM 2009.IEEE International Conference on, Edmonton, AB, 2009, pp. 73-82.[Fr11]Freitag, A., Matthes, F., Schulz, C., & Nowobilska, A. (2011). A method for business capability dependency analysis.In International Conference on IT-enabled Innovation in Enterprise (ICITIE2011), Sofia.[Geo13]Georges, P. M., & Hus, J. (2013). Six figure management method: How to grow your business with the only 6 KPIs you'll ever need. Kogan Page Publishers.[Jo03]Jollands, N. (2003). The usefulness of aggregate indicators in policy making and evaluation:a discussion with application to eco-efficiency indicatorsin New Zealand.[Ka13]Kandjani, H., Bernus, P., & Nielsen, S. (2013, January). Enterprise architecture cybernetics and the edge of chaos: Sustaining enterprises ascomplex systems in complex business environments. In System Sciences (HICSS), 2013 46th Hawaii International Conference on (pp. 3858-3867). IEEE.[Ki03]Kirchhof, “Ganzheitliches Komplexitatsmanagement,” Wiesbaden, Germany, Springer Fachmedien, 2003.[Lag13]Lagerstro m, C. Y. Baldwin, A. D. Maccormack, and S. Aier, “Visualizing and Measuring Enterprise Application Architecture: An Exploratory Telecom Case,”Harvard Business School Working Paper, no. 13-103, 2013.[La08]Lankes. Metrics for Application Landscapes: Status Quo, Development, and a Case Study: Dissertation. PhD thesis,Technische Universität München, München, 2008.[me12]mega, "Keys to Get a Simple, Organized View of your Application Landscape", 2012


References

22

[Mo09]Mocker, "What Is Complex About 273 Applications? Untangling Application Architecture Complexity in a Case of European Investment Banking,”System Sciences, 2009. HICSS '09. 42nd Hawaii International Conference on, Big Island, HI, 2009, pp. 1-14.[Mor02]Morisio, Maurizio and Stamelos, Ioannis and Tsoukias, Alexis, "A New Method to Evaluate Software Artifacts Against Predefined Profiles",SEKE '02 Proceedings of the 14th international conference on Software engineering and knowledge engineering, Pages 811-818[Po15]Podgórski, D. (2015). Measuring operational performance of OSH management system–A demonstration of AHP-based selection of leadingkey performance indicators. Safety science, 73, 146-166.[Re15]Rennenkampff, Management von IT-Agilitat, Entwicklung eines Kennzahlensystems zur Messung der Agilitat von Anwendungslandschaften,Dissertation. PhD thesis, Technische Universität Illmeanu, Illmenau, 2015[Ro03]Ross, J. W.: Creating a strategic IT Architecture Compentency: Learning in Stages. MIS Quarterly Executive, Vol. 2(1), 2003.[Ro06]Ross, P. Weill, and D.C. Robertson, “Enterprise Architecture As Strategy,” Boston, USA: Harvard Business School Press, 2006.[Sa08]Saqib, N., Siddiqi, M.T., 2008. Aggregation of safety performance indicators to higher-level indicators. Reliab. Eng. Syst. Saf. 93, 307–315. [Sc04]Scantlebury, W. Thiel, A. Datel, and S. Kimmel, “From IT Complexity to Commonality: Making Your Business More Nimble”,The Boston Consulting Group, 2004.[Schm13]Schmidt, C.; Widjaja, T.; Schutz, A.: Messung der Komplexitat von IT-Landschaften auf der Basis von Architektur-Metamodellen:Ein generischer Ansatz und dessen Anwendung im Rahmen der Architektur- Transformation. Informatik 2013 –Beitrage der 43. Jahrestagung der Gesellschaft fur Informatik. Kollen Verlag, Bonn.[Schn15]Schneider, A.W.; Reschenhofer, T.; Schütz, A. and Matthes, F.: Empirical Results for Application Landscape Complexity,48th Hawaii International Conference on System Science (HICSS), Kauai, USA, 2015[Schn16]Schneider, A. W. (2016): Decisions Support for Application Landscape Diversity Management (Doctoral dissertation, Technische Universität München)


References

23

[Ul11]Ulrich, W., & Rosen, M. (2011). The business capability map: the" rosetta stone" of business/it alignment. Cutter Consortium,Enterprise Architecture, 24(4).[Va05]Vasconcelos, A., Sousa, P., & Tribolet, J. (2005). Information System Architecture Evaluation: From Software to Enterprise Level Approaches.Paper presented at the Proceedings of the 12th European Conference on IT Evaluation (ECITE 2005).[We99]Weill, P., & Vitale, M. (1999). Assessing the health of an information systems applications portfolio: An example from process manufacturing.MIS quarterly, 601-624.


Application Landscape Evaluation Process

24

2

3

Identify Application Characteristics

Filter for relevant application characteristics

Screen relevant Application Characteristics

4

5

6

Develop aggregated KPIs to evaluate the AL of business capabilities based on the application characteristics

Conceptualization of AL evaluating KPIs

Evaluation of the developed KPIs by researcher and industrial experts

KPI evaluation

Gathering, Cleansing and Analysis of the Cooperation Partner’s Data

Data Collection and Cleansing

Individual AL assessment of each business capability by evaluating the included applications

AL Evaluation

1 Literature reviewIdentification of needed application characteristics to evaluate the application

Literature reviewExpert interview

USE

CAS

E

Expert interview

Literature reviewExperiments

7 Determine Recommendations for Action


Based on[We99, Sa06, Fa07, Kr09, Si10]

Research Approach

25

Identify problemand motivate

Define objectivesof a solution

Design &Development

Demonstration &Evaluation Discussion

Missing holistic view of AL status

Missing linkage between BCs and APM

Create a holistic view of AL status

Create a appropriate linkage of AL status and BCs

Developaggregated KPIs

Use business capability map to visualize AL status

Prototyping

Expert interviews

Design Science Approach

Process iteration


Based on [Peffers 2008, Hevner 2004]

Visualization Requirements

26

General requirements

Dashboard Time period Market KPI type Market comparability

Selection information

PowerPoint export

Capability visualization requirementsLayout Easy interpretation High-level perspective Strategic relevance

Application landscape visualization requirementsComplexity Quality Impact Application information

Application visualization requirementsApplication name Components information Transparency

Non-functional requirementsUsability Performance Maintainability


Visualization Logic

27

AL Complexity

Production

CapabilityMap APP 002

Product Design

Product Marketing

APP 019

APP 742

APP 867

APP 900

APP 019

APP 542

APP 003

Interface to 50 other applications- 5 within the same capability- 45 not in the same capability

Used in 7 Capabilities

5 years since go-live

10 underlying technology components


Data Model


final presentation master’s thesis: tool support for capability … · final presentation...

Documents