Framework Guidelines to Measure the Impact of BusinessIntelligence and Decision Support Methodologies in thePublic Sector

Roberto Boselli, Mirko Cesarini and Mario MezzanzanicaUniversity of Milan Bicocca, Milan, Italyroberto.boselli(unimib.ito.cesariniunimibitmezzanzanicaunimib.it

Abstract: Public Administrations started exploiting decision support systems (DSS) only in (very) recent times withrespect to the private sector where such systems have been used for long time to improve decision making activities(e.g. the DSS and Business Intelligence realm), Service efficiency and effectiveness improvement are the expectedresults of DSS exploitation, together with increased value for stakeholders. The adoption of DSS in the public sectorraises some questions: how to identify the areas where DSS introduction could greatly improve service quality andhow to measure the resulting added value? The paper will provide a literature review supporting the authors inidentifying the key factors influencing OSS value generation in the public sector. An evaluation model will besketched including a detailed set of dimensions. This paper aims at providing the ground for building an evaluationmethodology for assessing DSS adoption and exploitation in the context of public sector and public service provision.

Keywords: business intelligence, decision support systems, public administrations, public sector services, addedvalue measurement

1. IntroductionService delivery is among the most important functions of Governments and Public Administrations.Service delivery can be viewed as a value producing process for several stakeholders (Sanderson et al.2000). Service design and improvement issues have gained attention from the research community (e.g.Chesbrough and Spohrer 2006). Focusing on the public sector and services, several PublicAdministrations are adopting ICT to support and improve decision-making activities and active policiesenactment. This is done on the basis that methodologies, paradigms, and approaches developed in theframe of the Business Intelligence (BI) and Decision Support Systems (DSS) areas could greatly improvethe decision-making activities. Furthermore, the adoption of 81 and DSS in Public Administrations (PAs)is expected to start an enhancement process where information integration, data quality, anddevelopment of analytical and reporting models activities can be improved.

Some questions arise by focusing on the decision-making activities supported by ICT: which is the realadded value provided by these systems in the public sector? Can existing methodologies be used formeasuring the value generated by ICT in the public sector? Some literature works present models andmethodologies to calculate the ICT value in the private sector, for example they address issues such ashow to calculate the ICT impact on the organization’s processes. Can these approaches be smoothlyapplied to Public Administrations? Can the ICT impact on public organization processes be evaluated ina similar way as in the private sector?

These research questions stimulated the authors to define some framework guidelines trying to addressthese questions. The research presented in this paper focuses on 81 and DSS adoption in PAs, andwhether it is possible to identify and measure the Bl and DSS impact on PA processes, especially onknowledge production and sharing. Different levels of PAs can be identified, e.g. local regional or state(i.e. National), this paper focus on common considerations valid for both levels, although each level hassome peculiarities (e.g. extension of datasets, capabilities of funding DSS projects, involved processes,economy of scales) which have to be considered when applying the framework sketched in this paper.

ICT impact measurement is strictly related to the evaluation of public service performance, and both arechallenging research topics. 81 and DSS impact measurement is also challenging, and still poorly studiedespecially in relation to the public sector. This paper will make a contribution in this direction.

The novelty of our approach is in combining public sector services evaluation with 81 and DSS systemsadoption to improve public services performance. Issues such as the evaluation of service performancesor the 81 use in services are widely discussed in the literature, but the two issues have not been jointlyaddressed to the best of our knowledge. The public sector service performance is discussed in (Djellal

107

Roberto Boselli et aL

and GaNouj, 2008; Di Meglio et a). 2010; McAdam et aL, 2005) with focus on measuring tangibleperformance indicators. The evaluation of public sector Information Systems is discussed in (Newcomerand Caudle, 1991) where the authors focus on specific success indicators. DSS adoption to improveservice quality is addressed by (Pyon et al. 2009) but differences between the private and the publicsectors are not considered; while (Ramamurthy et a). 2007; Khan et al. 2010) consider only the privatesector when investigating the barriers and factors involved in DSS adoption. Another issue discussed inthis paper is the measurement of ICT impact on services. Works such as (Brynjolfsson, 1993; Bielowskiand Walczuch, 2002) consider the relation between service efficiency and ICT impact measuring outputson the basis of statistical and economic models, still with focus on private sector.

Furthermore, the paper introduces a framework to measure value, impact and adoption levels of SI in thepublic sector, on the basis of measurable indicators such as knowledge intensity, decision-makingintensity and automation degree. The paper investigates the relationships among the aforementionedmeasures and indicators in the context of public services, and such relationships have been poorlydiscussed in literature.

The paper is structured as follows: Sec. 2 will provide an overview of the differences between services inthe private and public sectors, Sec. 3 will provide some definitions of B) and DSS, and will analyze someB) measurement methods, Sec. 4 will focus on the adoption of B) in the PAs, Sec. 5 will introduce theframework to evaluate and classify public services, and finally Sec. 6 will draw the conclusions and willillustrate directions for future work.

2. The public sector servicesThe public sector provides several types of services, from health services, education, to social andcultural services, infrastructure, defence. A shared classification of public sector services (NACE)includes: public administration, defence and compulsory social security, education, health and socialwork, other community, social and personal services. Service research and literature provide severaldefinitions of the public sector, in this paper the authors consider the public sector to include allorganizations providing the aforementioned services.

2.1 Differences between private and public sectorsIt is worth to understand the PA5 characteristics and to focus on the differences between the public andthe private sectors in terms of objectives, information and knowledge utilization, and decision-makingprocesses. One obvious difference between the public and private sectors is that the public sector is notprofit driven and its primary goal is not to maximize profits (Røste and Miles 2005; Euske, 2003).Nevertheless, this should not lead to believe that public sector employees and managers are notconcerned about financial matters. Similarly to private companies, PAs fight for funding and power, andmainly for costs saving, but operate in a political environment and basically work to reach political goals(Murray, 1975). The PAs implement policies to provide benefits to the society as a whole, by deliveringbasic services to citizens that other organizations are not able to efficiently or equitably provide. To dothis, PAs have to meet objectives regarding productivity, efficiency and quality of services.

The way political goals are reached is influenced by the PA decision-making processes which are mainlyconditioned by the available information and the knowledge quality. Knowledge is essential to supportdecision-making activities (McAdam and Reid, 2001). Decision making and process activities are stronglybased on knowledge sharing and production processes that involve different actors, the service usersbeing the most important.

According to (Halvorson et al. 2005) PA services depend on revenues that are allocated according topolitical decisions rather than market performances. The central government funds public sector activitiesto cover the costs. The national budget makes public sector activities possible, and its allocation definesthe boundaries for public sector activities. Often public sector activities contents and scopes are far frombeing fully understood by citizens. Typically PAs do not specify in details how the funds are allocated andused. The next section will provide some answers on how PAs may improve their services.

2.2 Evaluation and improvement of public servicesCitizens demand for better services while supporting PAs services with their taxes (Langergaard andScheuer 2009). Therefore, two requirements deserve special attention among PAs: cost reduction andservice improvement, the latter involving concepts like service quality, effectiveness, and efficiency.

108

Roberto BoseIIi et a!.

Service improvement may require several actions: to modify the service processes, to improve theinformation quality, and eventually to carry out strategic knowledge management activities. Knowledge isa key factor in affecting PA service quality: knowledge is required to design, produce and deliver betterservices, furthermore knowledge may also represent the main output of some services.

Service quality improvement relies on evaluation, process and service evaluation requires useful andmeasurable indicators. As widely reported in the management literature, processes or services cannot beappropriately managed without measurements (Pyon 2009).

In the private sector efficiency and effectiveness measures are ultimately related to profit maximizationand to profitability for stakeholders. Therefore, in the private sector a classic performance metric is thereturn on investment (ROl) and the set of related indicators. However the public sector does not haveprofit maximization as main objective, but rather focuses on policy and service outcomes improvement.Unfortunately outcomes indicators are hard to identify since they are strictly domain dependent, and theyare affected by the complex set of factors influencing the customer perception and service satisfaction,both in short and long terms (Djellal and Gallouj, 2009).

Public services performance evaluation activities have been carried out only in recent times (Afonso,2006; Di Meglio et al. 2010). Two main approaches can be found in literature: the technical approachevaluates performances on the basis of productivity gains (Wölfl 2005; Kox and Rubalcaba 2007; Timmeret al. 2007); on the other hand, performances are evaluated according to management viewpoints(Osbourne and Gaebler 1992; Boland and Fowler 2000; Propper and Wilson 2003; de Brujin 2002).Service productivity measurement is a challenging issue in the service research. Measuring publicservice performances only on the basis of productive efficiency is undoubtedly a partial indicator ofoverall performance, on the other hand it is restrictive to consider only the economic indicators. Somescholars started adopting a more holistic perspective outlining innovation as a lever for improvement(Andersen and Corley, 2009). Therefore, the use of performance indicators in PAs has generatedinnovation demands and expectations in public service delivery processes.

PA5 should introduce innovation at different levels to improve services: organizational and administrativeinnovations, conceptual and policy innovations, innovations in service design processes, in the deliveryprocesses, and in the systems of interaction (Halvorson et al. 2005; Langergaard and Scheuer 2009).Innovation in the public sector is mainly driven by the need to improve governance and serviceperformance, including improved efficiency, in order to increase public value (Hartley 2005). ICT is beingpromoted within government and PAs as a means of improving the efficiency and effectiveness of servicedelivery to produce value for internal and external stakeholders (Sanderson et al. 2000; Beynon-Davies2007).

Several methodologies and paradigms are available in the literature to evaluate the added value providedby ICT in the service sector. Few of them focus on calculating the ICT value in the public sector andfewer on Bl and DSS. In the next sections the authors will show how BI and DSS can be used for PAsservices, and how the output of these systems can be evaluated for improving services.

3. BI and DSSAccording to (Golfarelli et al. 2004) “BI can be defined as the process of turning data into information andthen into knowledge [...] 81 was born within the industrial world in the early 90’s, to satisfy the managers’request for efficiently and effectively analyzing the enterprise data in order to better understand thesituation of their business and improving the decision process”.

According to (Lonnqvist and Pirttimãki 2006) BI has the purpose to aid in controlling the stocks and theflows of business information around and within the organizations by identifying and processing theinformation into condensed and useful managerial knowledge and intelligence. 81 presents businessinformation in a timely and easily consumed way and provides the ability to reason and understand themeaning behind business information through, for example, discovery, analysis, and ad hoc querying(Azoff and Charlesworth 2004). A BI system can be viewed as a DSS system focusing on data. Theterms will be alternatively used in this paper.

Different PAs have started projects for integrating the content of several administrative archives intocomprehensive repositories for statistical and analytical purposes, however the “81 portion” of the task

109

Roberto Boselli et al.

often lags behind. The delay of BI and OSS exploitation is only one of the differences between the publicand the private sector.

3.1 BI measurement methodsIn the BI literature several authors have identified 81 measurement as an important task (Solomon 2006;Viva 2000> but scholars agree that it is a difficult task to carry out (Gartz 2004; Hannula and Pirttimäki2003; Simon 1998). According to a recent survey only few organizations have any metrics in place for BIvalue measurement (Mann and Poulter 2004).

According to some works in literature (Popovic et al. 2010; Williams and Williams 2004; Lonnqvist andPirttimäki 2006) BI is an activity or a process like any other business process. Therefore, it is possible toapply business performance measurement methods to 81.

81 measurement serves two main purposes: first, to prove that it is worth the investment, and second tohelp managing the BI process, i.e. to ensure that the 81 products satisfy the users’ needs and that theprocess is efficient. Before describing methods for measuring 81 value, it is necessary to clarify theconcept of value in this context. From the point of view of a company using 81, the value is related toprofit improvement; while from the BI (end) user point of view, the value is somewhat related to perceivedusefulness. Any 81 value assessment needs to answer the following questions:

1) How much does 81 cost?

2) Which are the expected benefits of applying 81?

Calculating 81 costs requires calculating labour costs, software and hardware expenditure, externalinformation purchases, and other related expenses. 81 benefits measurement is not as simple asmeasuring the costs. Indeed, 81 provides mainly non-financial, intangible benefits such as improvedquality and timeliness information (Hannula and Pirttimäki 2003). ROl calculation is the typical method tomeasure an investment value, however the ‘81 outputs” (e.g. information and knowledge) are verydifficult to assess and quantify (Popovic et al. 2010). In literature (Davison 2001) proposed the ClMeasurement Model (CIMM) to calculate the ROl of a BI project. This model identifies various nonfinancial measures of strategic outputs useful to quantify the success of a 81 project, for example whetherthe targets set at the beginning of the project have been met, as well as the decision makers’ satisfaction.The limit of this model is that it is based mainly on qualitative assessments.

Shifting from the private to the public sector, measuring the value of 81 gets even more difficult forseveral reasons, firstly the lower importance given to profit and other financial indicators. Furthermore,the public sector is characterized by complex systems and multiple intangible variables which are difficultto measure.

Effectiveness and efficiency are considered among the main measures to assess the public sector. Theeffectiveness of 81 in public sector could be evaluated by exploiting the measures defined by (Herring1996) and (Sawka 2000) for the private sector. These measures could help investigating the decisionoutcomes while taking into consideration the public sector specificities. Namely, the 81 contribution couldbe evaluated by focusing on the specific decisions or actions (supported by the Bl) and then looking atthe benefit or detriment this decision brought to the related policy. This method identifies four paradigms:1) Bl can help in avoiding unnecessary costs, 2) decisions based on 81 processes may lead to enhancedrevenues (e.g., from taxes), 3) 81 information may help in improving resource allocation, and 4)identification of the direct link between a 81 decision and service performance. The 81 professional is theprincipal user of the information, therefore some of the most important BI measures focus on theefficiency of the personnel using 81, the resource allocation, the quality of the BI products and the usersatisfaction. The CIMM model is useful for this scope. Other methods to measure 81 performance, mainlyin the private sector are the Balanced Scorecard and the Performance Prism (Lonnqvist and Pirttimäki2006). Nevertheless they should be tailored to meet the public sector peculiarities.

4. BI adoption in PABI and DSS exploitation in the public sector is far behind the private one. Several reasons can be addedto explain this. (Nutt 2006) has investigated the differences between public and private decision-makingpractices. Some of the differences found can also be used to explain the aforementioned lag.

110

Roberto Boseili et a!.

a Private sector managers are more apt to support budget decisions made with analysis and less likelyto rely on bargaining. Public sector managers are less likely to support budget decisions backed byanalysis and more likely to support those that are derived from bargaining with agency people.

a Legislative mandates constrain budgets, in the past public sector leaders were limited or evenprohibited from spending money to collect information for decision-making. Many PAs wereprohibited from diverting funds from service delivery to collect data on emerging trends in servicedelivery. Even when information collection is now possible, professionals are reluctant to divertresources from service provision to collect such data.

a PAs have multiple goals, which can be vague, controversial, or both (Baker 1969; Bozeman 1984).Goal ambiguity makes vital performance outcomes unclear for public sector organizations.Although many of the reasons just introduced still hold in the PA, the pressure for obtaining knowledgeabout the population (and in real time), the need to offer better services with constrained resources havereduced the barriers for Bl exploitation. Furthermore the cost of the technologies necessary to implementa Bl/DSS project has diminished significantly in the past years, making the development of such projectsaffordable by all levels of the PA.

Therefore, Bl is playing more and more a key role in successful performance management initiativesbecause it allows managers to easily access up-to-date information and provide a comprehensive view ofwhat is happening in their area of responsibility. The information that BI provides aids decision-makingand helps civil servants monitoring and managing performances. Increasingly, public sector managersare using BI dashboards — visual displays that provide up-to-date indicators — and scorecards to trackperformance and budgets. In this way specific strategies can be defined and enacted by using a series ofmetrics and by setting thresholds that trigger alerts when they are exceeded.

In private organizations often the introduction of 81 has acted as a catalyst to improve the data qualityand to restructure the management processes, leading to big improvements in information accuracy andavailability. The same goal is pursued in public sector organizations where data quality is felt as a bigissue. Moreover, 81 strategies, technologies, and solution exploitations within the public sector lead tobetter business outcomes. For the past few years, BI has consistently ranked as a top priority forgovernment CIOs (Khan et al. 2010). Through collecting and analyzing data, BI creates detailed reportsthat provide inestimable insights. The benefits of these analyses are manifold; they can help bettermanaging an organization, improve performance and lower the cost of service delivery and so on. Insummary, BI can be useful in the public sector for several activities, including:a Measure, manage and report on performances

Policy formulation

a Planning and budgetinga Explore data hidden relationshipsa Disease surveillance and public healtha Identify tax fraud and money laundering• Homeland securitya Crime prevention

Moreover, BI technology has proved a useful application in many different areas of the public sector,including:a Financial Systems

• Acquisition, Logistics and Supply Chaina Health & Human Servicesa Citizen Relationship Managementa Knowledge I Case Managementa Intelligence Assessmenta Education & Campus ManagementPAs can apply 81 to improve their constituency’s knowledge, their ability to provide services and to obtainaccurate measurements of actions and policies effects. Moreover, PAs need to improve decision-making

111

Roberto Boselli et a!,

processes, and 81 tools and methodologies can enhance efficiency and performance, helping policy-leveldecision-making.

5. Framework guidelinesThe framework this paper proposes is based on the public service key features highlighted and on theconsiderations made in the past sections. Moreover, some specific models and frameworks in the serviceliterature influenced this work. One of them is the framework about knowledge and technologydimensions in the service sector (Kang 2006). Kang studied the different roles of technology andknowledge in services, and proposed a framework where services are classified in two categories:

Knowledge-embedded services, where the majority of knowledge is embedded in the serviceproduction system (i.e. the technology);• Knowledge-based services, where the majority of knowledge is held by the actors providing theservice (e.g. knowledge intensive business services - KIBS).The Kang framework is mainly aimed at classifying private sector services, but its logic can be applied tothe public sector.

The authors propose to integrate the Kang classifications with some other dimensions in order to build aframework useful for classifying public services, The resulting framework allows to evaluate (and to laythe ground for improvement of) ICT-based services focusing on the following aspects: cost savings,knowledge as value, improved policy and decision-making processes, data and information integration.The identified dimensions are: expenditure, knowledge intensity, decision-making intensity andautomation degree. The knowledge-intensity and the automation degree dimensions are drawn upon theKang framework logic. The knowledge-intensity evaluates the importance of knowledge within the servicewhile the automation degree evaluates how much ICT automates the information managementprocesses and conversely how much human intervention is required.

Services will be evaluated using a variable for each of the aforementioned dimensions. Some publicsector areas have been chosen to test the framework: administration, health, education and employmentservices. These areas provide knowledge intensive services and have different expenditure levels.Furthermore, the services of the same area may have different degrees of ICT-based automation anddifferent degrees of decision-making intensity.

The public sector services considered for the present framework (and showed later in the quadrants) are:a Administrative services, e.g. registry certifications;• Healthcare services;

• Vocational training services;

• Public employment services (PES).The services are showed in the quadrants of Figure 1 and Figure 2. The Health services arecharacterized by high expenditure and high knowledge intensity; administrative services are lessknowledge intensive but still have a high level of expenditure, while the other services have lowexpenditure and a middle degree of knowledge intensity. Figure 1 classifies the services according to theExpenditure and Knowledge intensity dimension.

In the second quadrant (Figure 2) the same services are classified with the other two dimensions, namelydegree of (ICT-based) automation and degree of decision-making intensity. According to thesedimensions health services, PES and vocational training services have low level of automation, whilehealth services have higher decision intensity than the others. Only the administrative services show ahigh degree of automation and low decision intensity.

The development of a 81 or a DSS system is a very resource consuming task, DSS and BI projects areon/off investments: they return positive results (i.e. they provide value to the decision-making activities)only if the decision maker’s needs are correctly identified, useful indicators and measures are computed,data quality issues are resolved, the technological support is correctly deployed, the data provisionsystem is user-friendly and affects the decision-making process and the overall service provisioningprocess. Should only one of this aspects not being properly managed, the resulting decision supportsystem will fail to provide an added value to its users.

112

High

Ca,a>

uJ

Roberto Boselli et al.

Low Knowledge intensity

Figure 1: Knowledge and expenditure dimensions

Figure 2: Decision and automation dimensions

High

The costs and the probability of failure are lower when prior knowledge about the domain and the projectare available among the users and the ICT personnel involved in the project (e.g. because peoplealready worked on similar contexts). However, fewer successful projects are available in the public sectorcompared to the private, because of the aforementioned lack of DSS diffusion among the public sector.For these reasons, it can be suggested to start DSS projects in the public domain where the probability offailure is low and where the expected benefits could be very high. The dimensions and the quadrantsintroduced in this section help identifying the PA sectors where DSS projects could provide tangibleresults lowering at the same time the probability to fail (and consequently to waste public funds). NamelyPA sectors (or services) having high knowledge intensity could benefit from the introduction of DSSsystems, furthermore the decision-making activities would benefit from the introduction of BI systems (i.e.DSS systems focusing on data). The introduction of DSS systems could lead to huge savings in sectorshaving high expenditures, or could lead to a service level improvement without cost changes. Services orsector having a high degree of decision will have a relief from the introduction of DSS systems, while ahigh level of automation is an indicator of the availability of electronic data upon which the DSS can bebuilt. Where a tot of electronic data is available, a lot of useful information can be identified and extractedwith low effort. Thus a high level of automation may contribute to lower the costs (and the risk of failure)of a DSS project.

6. ConclusionsThe research presented in this paper focuses on Bl and DSS adoption in PAs, and whether it is possibleto evaluate the BI and DSS impact on knowledge production and sharing processes within PAs. A

Health servicesAdministrativeservices

Vocational tra ing services

PESs rvices

Low

High

C0

c)a,

Low

Health services

PES services

Vocational trainingservices

Administrativeservices

Low Automation High

113

Roberto Boselli et a!.

literature survey on public sector services helped to identify available service measurement andevaluation methodologies. With the analysis of 81 and DSS adoption in PA5 it was possible to identifysome drivers and motivations to use BI in public services, and also to identify some dimensions whichare useful to classify public services. These dimensions shape an initial framework to classify publicservices.

Moreover, the framework should also help to identify dimensions of analysis to assess the BI impact inpublic services. The quadrants proposed in the paper will allow to create a map of all public sectorservices, and to identify areas where SI adoption could be effective in improving the service efficiencyand effectiveness. 81 and DSS are especially useful in areas having complex service productionprocesses (e.g. Healthcare sector). These areas are characterized by high expenditures and by highknowledge intensity and decision intensity degrees. In future works the framework will be tested withempirical data collected in the frame of public service case studies. The proposed framework dimensionshave to be enriched with indicators about, for example, user satisfaction, policies outcome and otherrelated topics.

ReferencesAfonso, A., Schuknecht, L. and Tanzi, V. (2006) “Public Sector Efficiency: Evidence for New EU Member States andEmerging Markets”, Working Paper Series, No. 581.Andersen B. and M.Corley (2009), Productivity and the Service Sector: Theories, Concepts and Measurements,Edward Elgar.Azoff, M. and Charlesworth, I. (2004) The New Business Intelligence. A European Perspective, Butler Group, WhitePaper.Baker, R. (1969) “Organizational theory in the public sector”, Journal of Management Studies, Vol. 6, No.1.Beynon-Davies, P. (2007) “Models for eGovernment”, Transforming Government: people, process and policy, Vol. 1,No. 1.Bielowski, A.G. and Walczuch, R. (2002) “ICT-lmpact on Services”, Proceedings of the 35th Hawaii InternationalConference on System Sciences, vol. 8, pp.259-268.Boland, T. and Fowler, A. (2000) “A systems perspective of performance management in public sectororganisations”, The International Journal of Public Sector Management, Vol. 13, No. 5, pp 41 7-446.Bozeman, B. (1984) “Dimensions of “publicness”: An approach to public organization theory”, New directions inpublic administration, Vol. 46.Brynjoifsson, E. (1993). “The productMty paradox of information technology”, Communication of the ACM, Vol. 36,No.12. pp. 67-77.Chesbrough, H. and Spohrer, J. (2006). “A research manifesto for services science’, Communication of the ACM,Vol. 49, Num. 7.Davison, L. (2001) “Measuring Competitive Intelligence Effectiveness: Insights from the Advertising Industry”,Competitive Intelligence Review, Vol. 12, No. 4.De Brujin H. (2002) “Performance measurement in the public sector: strategies to cope with the risks of performancemeasurement’, The International Journal of Publlc Sector Management, Vol. 15, No. 7, pp 578-594,Di Meglio G., Stare M., Maroto A. and Rubalcaba L. (2010) “Public services in the enlarged EU: An assessment ofperformance and efficiency”, Proceedings of RESER2O1O, Goteborg, Sweden.Djeilal, F. and Gallouj, F. (2008) Measuring and improving productivity in services: Issues, strategies and challenges,Edward Elgar, London.Djellal, F. and Gallouj, F. (2009) “Public services and the productivity challenge”, Deliverable 2.1-A2.1, ServPPlNProject.Euske, K.J. (2003), “Public, private, not-for-profit: everybody is unique?”, Measuring Business Excellence, Vol. 7 No.4, pp. 5-11.Gartz, U. (2004) “Enterprise Information Management”, in Raisinghani, M. (Ed.) Business Intelligence in the DigitalEconomy: Opportunities, Limitations and Risks, Idea Group Publishing, Hershey, PA.Golfarelli, M., Rizzi, S. and Celia, I. (2004) “Beyond data warehousing: what’s next in business intelligence?”,Proceedings of the 7th ACM international workshop on Data warehousing and OLAP, ACM.Halvorson, T., Hauknes, J., Miles I. and Røste, R. (2005) On the Differences Between public and private sectorinnovation, Public Report No. 9, NIFU STEP.Hannula, M. and Pirttimãki, V. (2003) “Business Intelligence - Empirical Study on the Top 50 Finnish Companies”,Journal of American Academy of Business, Vol. 2, No. 2, pp 593-599.Hartley, J. (2005) “Innovation in Governance and Public Services: Past and Present”, Public Money andManagement, pp. 27-34.Herring, J. (1996) “Measuring the Value of Competitive Intelligence: Accessing and Communicating Cl’s Value toYour Organization”, SCIP Monograph Series, Alexandria, VA.Khan, A.M.A., Amin, N. and Lambrou, N. (2010) “Drivers and Barriers to Business Intelligence Adoption: A Case ofPakistan”, Proceedings of the European and Mediterranean Conference on Information Systems (EMCIS2O1O),Abu Dhabi, UAE.Kang, H. (2006) “Technology management in services: knowledge-based vs. knowledge-embedded services”,Strategic Change, Vol. 15, No.2, pp 67-74.

114

Roberto Boselli et a!.

Kox, H. and Rubalcaba, L. (2007> “The Contribution of Business Services to European Economic Growth”, inRubalcaba and Kox (ed.) Business Services in European Economic Growth, Palgrave/Macmillan, pp 74-94.Langergaard, Li. and Scheuer, J.D. (2009) Speciflcities of public sector service innovation, ServPPIN Deliverable2.1.

Lonnqvist, A. and Pirttimäki, V. (2006) ‘The measurement of business intelligence”, Information SystemsManagement, Vol. 23, No. 1.

Mann, J. and Poulter, A. (2004) “Dissemination of Competitive Intelligence”, Journal of Information Science, Vol. 30,No. 2, pp 165-180.McAdam, R. and Reid, R. (2001), “A comparison of public and private sector perceptions and use of knowledge

management”, Journal of European Industrial Training, Vol. 24, No. 6, p. 317.McAdam, R., Hazlett, S.A. and Casey, C. (2005) ‘Performance management in the UK public sector”, InternationalJournal of Public Sector Management, Vol. 18, No. 3, pp. 256-273.Murray, M.A. (1975) “Comparing Public and Private Management: An Exploratory Essay’, Public Administration

Review, Vol. 35, No, 4 pp. 364-371,NACE, Statistical Classification of Economic Activities in the European Community, [online]

http/!ec.europa.eulcsmpetition/merger&casesiindexInaceailhtmLNewcomer, K. E. and Caudle, S. L. (1991) “Evaluating Public Sector Information Systems: More Than Meets theEye”, Public Administration Review, Vol. 51, No. 5, pp. 377-384.Nutt, P. (2006) “Comparing public and private sector decision-making practices”, Journal of Public AdministrationResearch and Theory, Vol. 16, No. 2.Osbourne, D. and Gaebler, T. (1992) Rehventing Government, Addison-Wesley.Popovic, A., Turk, T. and Jaklic, J. (2010) “Conceptual Model of Business Value of Business Intelligence Systems”,Journal of Contemporary Management Issues, Vol.15, No.1,Propper, C. and Wilson, D. (2003) “The use and usefulness of performance measures in the public sector”, CMPOWorking Paper Series Vol. 03, No. 73.Pyon, C.U., Lee, M.J, and Park, S.C. (2009) “Decision support system for service quality management usingcustomer knowledge in public service organization”, Expert Systems with Applications, 36, pp. 8227-8238.Ramamurthy, K.R. (2007) “An empirical investigation of the key determinants of data warehouse adoption”, DecisionSupport Systems, Vol. 44, No. 4, pp 8 17-841.Røste, R. and Miles, I. (2005) “Difference between public and private sector innovation”, Publin Report.Sanderson, I., Bovaird, T., Davies, P., Martin, S. and Foreman, A. (2000) “Made to Measure: Evaluation in Practicein Local Government”, Local Government Management Board, London.Sawka, K. (2000) “Are We Valuable?”, Competitive Intelligence Magazine, Vol. 3, No. 2.Simon, N.J., (1998) “Determining Measures of Success”, Competitive Intelligence Magazine, Vol. 1, No. 2, pp 45-48.Solomon, M. (1996> “The Intelligence Asset-Building Process”, Competitive Intelligence Review, Vol. 7, N. 4, pp 69-76.Timmer, P.M., O’Mahony M. and van Ark, B. (2007) “EUKLEMS Growth and Productivity Accounts: Overview”,[online] www.euklems.net,Viva Business Intelligence Inc. (2000) “Measuring the Benefits of Business Intelligence”, Pro-How Paper, Vol. 2.Williams, S. and Williams, N. (2004) “Capturing ROl through Business-Centric BI Development Methods”, DMReview, August.WbIfl, A. (2005) “The service economy in OECD countries”, STI Working Paper, Vol. 3.

115