ict impact indicators: linking data from different sources partnership event, geneva wednesday may...
TRANSCRIPT
ICT IMPACT INDICATORS:LINKING DATA FROM DIFFERENT SOURCES
Partnership Event, GenevaWednesday May 28th 2008
An EU initiative with 13 National Statistics Offices
Tony ClaytonUK Office for National Statistics
Developing Impact Indicators
Agenda
• Background to ICT indicator development• Limits to surveys – what we actually measure• Impact assessment; evidence from firms• Using firm level data to develop indicators –
the advantages• …. and the challenges to overcome
Background to ICT indicators
Level ofElectronicCommerceActivity
Time
Readiness
Intensity
Impact
OECD ‘diffusion’ model, based on linear view
• Readiness; ability of firms, individuals to adopt technology
• Intensity / use; measures of the proportion of firms, individuals who adopt, and the amount of use
• Impact; changes in behaviour, and in economic structure and performance as result of use
Background to EU ICT indicators
E- Europe indicators to 2005 focused on use• Citizens
- access or ownership of technology
- use of internet, and applications
• Enterprises- access to or ownership of technology
- investment in networks / availability of broadband
- use of internet and e-business links
• Government- services available on line and degree of take-up
- investment in specific areas, e.g. education
• Impact?- Only e-commerce, and some individual metrics, to represent
changed behaviour resulting from use
Background to EU ICT indicators
i2010 more focused on impacts
• EU Information space- development of broadband availability, take-up, speed and pricing
- multiplatform access in firms
• Advanced services- focused on household / individual uses of range of internet services
• Security- incidence of problems and solutions
- attitudes to risk / trust and confidence
• Impact- ICT investment, use and impact on growth / productivity
• ICT innovation- ICT R&D and effects on business transformation
Background to ICT indicators
‘Impacts’ now need to reflect innovation in ICT use
• Innovation no longer a linear process- depends on networks / feedback / co-creation
• ICT as ‘general purpose technology’- major impacts on economic performance based on use
- dependent on range of complementary investments
• Innovative uses particularly strong in services- majority of ICT investment in market services
- returns to IT investment strongest in services
- .. where growth, productivity and innovation are hardest to measure
• Pervasive presence of ICT- impacts too important not to be quantified, as part of economic
measurement
Limits to surveys
Measuring ICT investment is hard to do
• Much existing data doesn’t reflect ICT properly as capital- macroeconomic treatment of software investment is inconsistent
between countries
- firm level source data misses out large parts of ‘own account’ IT software investment – because the respondents can’t quantify it
- as ICT service purchases, and IT enabled services, substitute for ‘own investment’, can we pick up the effects?
- (Finland results for this study suggests productivity gains from IT outsourcing will ensure this grows)
• Measurement differences can be very significant …..
Limits to surveys … and macro data
0 0.5 1 1.5 2 2.5
Sweden
Britain 'New'
United States
Finland
Australia*
Japan*
Germany
France
Britain 'Old'
Software investment as % GDP 2003
* 2002
Economist Feb 2006
Limits to surveys
What else have we learned about indicators?
• Surveys about ‘self assessed’ impacts not reliable- assessment of payback from ICT can be inconsistent
- links to actual outcomes from production surveys do a better job
• Questions on technology less useful than what is done with it- technology changes, and respondents don’t always understand it
- effects on behaviour may be a better guide to economic effects
• ICT indicators are sensitive to structural differences- industry characteristics can be stronger than national variations
- member state estimates may not be comparable, in spite of similar methods.
Who we are
• 8 NSIs directly involved in developing research, metadata and method
- UK, Sweden, Finland, Denmark, Italy, France, Austria, Netherlands
Slovenia involved through Ljubljana University
• 4 more joined to use method and contribute to analysis
- Germany, Norway, Czech Republic and Ireland
• Academic support from Free University of Amsterdam
What we’ve done
• Firm level analysis within countries, linking surveys on ICT use, output, productivity, IT investment, skills, innovation …
Core variables from common EU ICT use survey‘Add-on’ analysis in countries with extra survey data
• Built metadata system to aggregate data for industries / countries consistently
Country / industry indicators built using survey dataAble to bring in other data from National AccountsAnalysis across different environmentsNew method for indicators
From firm data to macro indicators
What we’ve done …..
• Firm level analysis within countries, linking surveys on ICT use, output, etc to identify …- core variables from common EU ICT use survey
- ‘add-on’ analysis in countries with extra survey data
• Built metadata framework to aggregate this data for industries / countries on consistent basis- country / industry indicators built using survey data
- able to bring in other data from National Accounts
- analysis across different environments
- new method for constructing indicators
From firm data to macro indicators
Industry / country aggregates help
• We can aggregate to comparable intermediate aggregate data- metadata used to generate completely comparable aggregated data by
country / industry / firm size / ownership / market characteristics etc..
- then we can analyse at this level
• From ‘industry / country’ analysis we get- ability to do analysis across different national policy environments.
- ability to bring in data not available at firm level
- quantification of effects at macro level – £billion more persuasive than a regression coefficient!
Res
earc
her
Policy QuestionResearch Design Program Code
Publication
Net
wor
k Metadata
Networkmembers
Cross-countryTables
NS
Os
Provision of metadata.Approval of access.Disclosure analysis
From firm data to macro indicators
• Assemble the data- put production survey, ICT use survey and business register datasets
in single research environment
• Construct the metadata- describe each survey in standard terms; variable names, format,
range,
- test linkages between units in surveys using register identifiers
• Run analytical code on datasets, using metadata to access and interpret
- code uses SAS or STATA operating within NSI
- produces results / aggregates which can be shared
- disclosure checking before use (can build into code)
What NSIs have to do to join …..
From firm data to macro indicators
ICT metrics
Variables for most countries ‘Add-on’ variablesFirm uses computers
% workers using computers
Firm uses internet
% of workers with access to internet
Firm has website
Firm orders via internet (or EDI etc)
% of orders (or value)
Firm sells via internet (or EDI etc)
% of sales (or value)
Firm has intranet
% of workers with access to intranet
Firm has fast internet access
% of workers with access to fast internet access
Use of electronic business processes
Mobility of access to technology
Mobility of workers accessing technology
IT investment (or expenditure) at firm level
From firm data to macro indicators
Contextual variables / complements to ITVariables for most countries ‘Add-on’ VariablesFirm size
Firm age
Employment
Payroll
Capital Stock
Ownership (local or international)
Outsource ratios (measured as purchases / sales or similar)
Measures of labour ‘quality’ (measured in terms of pay per person)
Industry characteristics (growth, entry / exit, firm share turbulence)
Multinational organisation of firms (from FDI and ownership data)
Employee skills (from linked employer – employee datasets)
Changes in organisation (from some local surveys, or CIS)
Measures of innovation (from CIS)
From firm data to macro indicators
Measures of Impact
Levels Growth ratesProductivity measured as- Gross sales / employee (vs industry)- V.A. / employee (vs industry)
Productivity measured in terms of Multi-Factor Productivity (for countries which have firm level data on investment, IT investment, labour and other inputs)
Productivity change measured as- Gross sales / employee (vs industry)- V.A. / employee (vs industry)
Growth rates measured for:- Gross sales (deflated by industry PPI)- Market share of IT using firms in industry- Value added (double deflated where possible)
- Employment, or share of employment in industry
From firm data to macro indicators
country Elecom MexElec DISTR FINBU TOT
SWE7 11 9 10 11
NOR9 10 8 9 10
FIN8 9 10 8 9
GBR6 8 5 4 8
NLD4 6 4 6 7
SLO5 7 7 6
GER7 5
CZE3 4 6 5 4
AUT5 3 3 3 3
ITA1 2 2 1 2
IRE2 1 1 2 1
From firm data to macro indicators
…and what it can deliver (cont)(example; index of fast-internet enabled workers)
From firm data to macro indicators
…and what it can deliver (cont)(example; index of purchases made through the internet)
country Elecom MexElec DISTR FINBU TOTNOR
10 12 11 11 12
FRA9 10 9 7 11
AUT8 9 10 8 10
GER11 9
SWE7 7 3 10 8
FIN4 4 7 4 7
CZE5 8 6 9 6
IRE6 5 8 5 5
GBR3 6 4 6 4
NLD2 3 5 3 3
ITA1 2 2 1 2
SLO. 1 1 2 1
From firm data to macro indicators
…and what it can deliver (cont)(example; index of sales made through the internet)
country Elecom MexElec DISTR FINBU TOTNOR
10 12 11 11 12
AUT9 11 10 10 11
FRA8 10 8 9 10
GER. 9 . . 9
CZE4 8 6 5 8
FIN7 7 7 4 7
IRE6 5 5 7 6
NLD5 4 9 6 5
GBR1 6 4 2 4
SWE3 2 3 8 3
ITA2 3 2 1 2
SLO. 1 1 3 1
Impact evidence from firms
What microdata contributes to indicators
• From micro analysis we get- clear evidence on how inputs to innovation affect behaviour, competition
and performance at level where decisions are made
- pointers to policy ’levers to pull’ in terms of firm level incentives
- limited ability to compare drivers of performnce between countries
• Advantages and disadvantages- firm level analysis is flexible, fast, and uses ‘all the data’ in terms of
variation / degrees of freedom
- but international comparisons of microdata analysis are vulnerable to differences in surveys / definitions / data treatments
Impact evidence from firms
Earlier firm level analysis showed …..
• On productivity impact - IT investment and use associated with higher productivity measured
through labour productivity or MFP (for a limited number of countries)
- Productivity of IT effects stronger in US owned firms, in multinationals, in younger and more innovative firms, and in services
• On IT investment and CT deployment- links between communications infrastructure and firm investment IT are
significant (for UK at least)
- broadband using workers have a significant separate effect on productivity, over and above IT investment
Impact evidence from firms
What this study has shown from firm level analysis includes …..
• On productivity impact - consistency across 12 countries in productivity effects of core ICT use
metrics for manufacturing firms
- much more difference in services; UK NL, France and Scandinavian analyses show significant positive effects, elsewhere mixed picture
• On IT investment and CT deployment- broadband is a good indicator – or predictor - of firm level purchased
investment
- broadband using workers still have a significant separate effect on productivity in countries where use is greatest
- ‘joining up workers’ has the biggest effect in differentiated services, where much of the investment is in systems and people
Impact evidence from firms
What firm level analysis has shown
continued …..
• On Innovation - evidence that connected workers are more likely to innovate
- .. From UK that firms which get innovative ideas from third parties are more likely to invest in broadband connections
- .. From NL / SE that ICT using firms are more likely to produce innovation outputs, given other inputs, and that productivity effects associated with ICT are strongly influenced by this
- => it’s what firms change with ICT that counts, not what they spend
• On ICT and skills- higher skills levels increase returns from ICT investment, or from ICT
use, whether we use direct measures of skills, or wage measures
Impact evidence from firms
What we’ve shown from firm level analysis continued …..
• On organisation - confirmation that some business process links can be associated with
significant, positive, productivity effects, in some countries
- some of the organisation / maturity variables can be expressed in terms of a composite index … but our surveys need more work
• On flexibility / specialisation- Finnish evidence that flexibility associated with mobile access to IT has
significant productivity benefits
- Finnish evidence that outsourcing of IT services is associated with worthwhile labour productivity advantages
Impact evidence from new indicators
What we show from industry / country level analysis includes…..
• On ICT use- Evidence that broadband connected employees provide a strong
productivity indicator across the contributing countries, over the period 2003-2005
- Looks to be significant across industries, even though firm level effects are uneven
• On ICT use as driver of competition- Cross – EU evidence that higher ICT use, and productivity effects is
associated with more dynamic competition
- Consistent with US conclusion by Brynjolfson et al, that ICT ‘enterprise architecture’ enables firms to replicate successful business models
Country / industry data shows broadband gains:
100
200
300
400
500
600
0 10 20 30 40 50 60 70 80 90 100Labo
ur p
rodu
ctivi
ty in
dex
acro
ss s
ecto
rs (b
ase=
100)
% of employees linked by broadband
Labour productivity and broadband in selected EU countries (2001-04)
..and industry split shows differential impacts :
…..industry splits tend to support UK microdata results showing- IT capital matters most in manufacturing- Broadband enabled employees matter more in differentiated services
Electrical and Electronic Equipment
0
20
40
60
80
100
0 20 40 60 80 100
% of workers with access to broadband
Ho
url
y P
rod
uct
ivit
y
Paper and Publishing
0
20
40
60
80
100
0 20 40 60 80 100
% of workers with access to broadband
Ho
url
y P
rod
uct
ivit
y
Financial Services
0
20
40
60
80
100
0 20 40 60 80 100
% of workers with access to broadband
Ho
url
y P
rod
uct
ivit
y
Advantages
Micro based macro indicators can:
• Use data available at both firm and industry level - industry indicators compiled ‘bottom up’ from linked surveys
- combined with sources from National Accounts or Labour Market
- aim to include innovation indicators in next round of work
• Build comparisons across countries- firm level data can’t be shared by most NSIs, and are different
- metadata approach can iron out differences before analysis
• Allow for structural differences between countries- disaggregation possible on completely comparable basis
- only need to write analytical code once
• It works!- all 13 countries in this project have produced outputs, despite very
different systems
Challenges
Micro based macro indicators need:
• Initial investment in metadata and systems - work to set out dataset and variable descriptions from key surveys
- must be able to bring datasets together within each NSI in a common research environment, to enable them to be linked
• Systems to control confidentiality within each NSI- research / linking framework must meet national non-disclosure
requirements, but still allow firm level data to generate required outputs
• Agreement on confidentiality controls for merged dataset- agreement across NSIs on the level of detail which can be used for
research and analysis, even if it doesn’t meet national quality standards
- agreement on what detail can go into the public domain.
• Persistence!- Work in this project produced lots of data which needs simplifying!