workshop in montreal

Elie Geisler

Professor and Associate Dean for ResearchStuart Graduate School of BusinessIllinois Institute of TechnologyChicago, Illinoisemail: [email protected] June 17, 2004

Prepared for Presentation at the “Workshop on Measuring the Impacts of Science,” Montreal, Canada, June 16-18, 2004

JUNE

2004

TERMINOLOGY

Public-Sector Science & Technology (S&T)

Impacts on Industry

Impacts on Society

Commercialization of S&T

Technology Transfer from Federal Laboratories

WORKSHOP

IN

MONTREAL

METHODOLOGIES

STATISTICAL, CO-VARIATION DESIGN

OPENING THE BOX

Comparing databases distinct in time and phenomenon (ontology): temporal & conceptual gaps

Ex post factor Explanation as Substitute for Strong Theory

Process Approach Linkages explored between stages Monitoring transformations of

constructs

WORKSHOP

IN

MONTREAL

WHAT DO WE STUDY?

PROCESSES: How public sector laboratories conduct their activities? Issues of efficiency & effectiveness.

OUTCOMES

• How successful are public-sector laboratories

• Success: Defined as commercialization and technology transfer

• Success: Defined as contributions to parent agency

IMPACTS on industry & society

WORKSHOP

IN

MONTREAL

What are the stages of transformation of public R&D into technology transferred and commercialized?

How can we measure such a process: What are the metrics?

A MODEL OF OUTCOMES AND IMPACTS

WORKSHOP

IN

MONTREAL

Cultural variables(sector, industry,

company, orGovernment agency)

Available poolof metrics

Management policies,strategy, and biases

Prior experience withevaluator and metrics

Perceptions of thevalue and effectiveness

of metric

How managers perceivethe link between S&T

and the strategic objectives

of the organization

Type of activity to bemeasured

Customers’ choicesand preferences

Interests and preferencesof other stakeholders

Development ofSelection Criteria

& Selection Process

A GENERALIZED MODEL OF METRIC SELECTION

Impacts on behavior of managers regarding S&T

Impacts on the organization(effects on the strategy and culture)

Feedback

WORKSHOP

IN

MONTREAL

KEY ISSUES IN METHODOLOGYAND ANALYSIS OF SCIENCE INDICATORS

CATEGORY KEY ISSUES

Methodology:Selection and

Structureof Indicators

Lack of a unifying theory or conceptual scheme

Influence of goals, motives, and biases of evaluators

Problem inherent in output indicators Preference for available data Difficulties with convergence and macro

indicators

Analysis and Interpretation

Do these indicators really represent the state and progress of science? If so, to what extent?

“Leap of faith” from the indicators to policy conclusions.

Manipulations, correlations, and indices may lead to erroneous trends and conclusions.

Indicators selected that are based on distinct theories (e.g., economic theories) may lead to conclusions biased by these theories.

WORKSHOP

IN

MONTREAL

ISSUES IN THE USE OF INDICATORS TO MEASURE COMPLEXPHENOMENA AND CONCEPTUAL CONSTRUCTS

QUERY DEFINITION OF ISSUES

How well is the construct defined? o Using convention to assign indicators to ill-defined constructs

o In such ill-defined constructs, do the indicators assigned adequately measure?

How many are adequate? o By using multiple indicators, under which criteria and conditions is a given number enough to adequately measure the constructs?

Can the indicators converge and be indexed

or aggregated?

o Do indicators converge by the dimensions they purport to measure?

o Can indicators be aggregated into “Mono-Indicators?”

o How to deal with indicators that measure surrogate variables, aspects, or dimensions of the concept? Can they be aggregated with “direct” indicators?

Are there biases and distortions in

measurements?

o Choice of indicators is a function of value judgments and social, political, and other agenda.

o Some indicators have more power than other indicators in a given set, thus distorting the measurement in their favor.

o Indicators may be poorly correlated in a given set, hindering convergence and adding to biases.

o The mere choice of indicators may affect the behavior of the phenomena we are measuring.

o Errors and biases in statistical computations.

WORKSHOP

IN

MONTREAL

The Process-Outcomes Model of the Linkages Between the R&D Process and Social and Economic Systems

Inputs to R&D(in particularR&D sector orthe total national S&T system), e.g.

People & skillsFundingGuidance & gainOther resourcesand restraints

The R&Dprocess, e.g.

EnergyHealth careIndustryEducation

Immediate ordirect outputsfrom R&D/science (potentialinputs to socialsubsystems), e.g.,

PublicationsPatentsIdeasTheoriesDiscoveriesMethods

1 2

A

Other factors in the specific situation (e.g., the R&D sector or the social subsystem) which affect the transition, transportation, adoption, usefulness, cost or economic benefit from transfers between adjacent and more distant stages. Such factors may include economic, cultural, organizational, technical, personal, or political ones. Some are particular to a given stage (e.g., the barriers and difficulties involved in designing economical and socially acceptable energy or safety devices and systems or the diffusion problems in curing a disease); others may apply to several stages in the overall process (e.g., capital shortages or regulations); and still others are pervasive across the whole process (e.g., organizational barriers to innovation, individual risk preferences, diffuse decision-making responsibility).

Transformation,and diffusionprocesses inthe economic &social sub-systems, e.g.,

Health care deliveryTransportation servicesEnergy systemConsumer or industrial products & services

Intermediateoutputs of R&Dfactual inputsto social sub-systems, e.g.,

New and improved products or processes Methods of organizing, managing, or evaluating

B

3

C


Transformationand diffusionprocesses, e.g.,

MarketingAdoptionDesignImplementation

Pre-ultimate R&D(outputs of social/economic subsystems),e.g.,

Mortality and morbidityExtinction of particular causes of deathImproved safety of products and work environmentsProductivity rates in individual firms or sectors

Transformationand diffusionprocesses in thesociety and theeconomy

Ultimate R&D outputs(quality of life andhealth of the economyand society), e.g.,

International balance of tradeEnergy independenceGross national productComponents of quality of life

4 5

D


METRICS

CORE indicators of outputs/outcomes from public-sector laboratories.

ORGANIZATION-SPECIFIC indicators of outputs/outcomes from public-sector laboratories

WORKSHOP

IN

MONTREAL

ILLUSTRATIVE CORE INDICATORS AND MEASURES OF THEOUTPUTS FROM R&D

I. IMMEDIATE OUTPUTS1. Written Scientific and Technical Outputs

1.1 Number of publications in refereed journals.1.2 Number of technical reports.1.3 Number of patents.1.4 Number of citations in refereed journals.1.5 Number of patent disclosures.

2. Other Outputs2.1 Number of licenses signed for own patents.2.2 Number of new products conceived2.3 Number of key improvements suggested.2.4 Number of new and improved test methods, models, standards, concepts, and databases.2.5 Number of new ideas transferred downstream.2.6 Number of problems solved for users/clients.

3. Overall Reputation 3.1 Number of complaints by clients/users. 3.2 Judgment of quality of R&D by clients/users. 3.3 Number of awards received. 3.4 Milestones/objectives met.

WORKSHOP

IN

MONTREAL

1. Scientific/Technical Impacts on Direct Users of R&D 1.1 Number of improved or new products produced. 1.2 Number of improved or new processes applied. 1.3 Number of improved or new materials made.

2. Economic/Financial Impacts on Direct Users of R&D 2.1 Income from licensing patents and inventions. 2.2 Cost reductions/savings from new and improved products/processes 2.3 Improvements in productivity of people and

materials/equipment. 2.4 Costs associated with implementation and adoption of

new products/processes (e.g., training, regulatory adjustments).

3. Responsiveness of R&D 3.1 Judgment by direct clients/users (their satisfaction; benefits from R&D). 3.2. Judgment by other organizations (R&D and non-R&D).

II. INTERMEDIATE OUTPUTS

WORKSHOP

IN

MONTREAL

III. PRE-ULTIMATE OUTPUTS

1. Economic Benefits and Costs1.1 Improvements in productivity levels in people and equipment by sector and industry.1.2 Savings, cost reductions, and income generated by improved health, productivity, safety, and mobility of the workforce at sectoral and national levels.1.3 Costs to the economy and to society from the absorption of R&D into social/economic subsystems (such as transportation, energy, information, telecommunications, and health care).

2. Improvements and Problems in Social Conditions 2.1 Improvements in overall health of population.2.2 Improved life expectancy of population.2.3 Improved satisfaction and levels of optimism of the population.2.4 Problems in social conditions (associated with the pre- ultimate outputs and their impacts on society and the economy) such as increased alienation, decrease in feelings of security and job loyalty.2.5 Changes (positive and negative) in the nature of work.

WORKSHOP

IN

MONTREAL

IV. ULTIMATE OUTPUTS

Source for some of the indicators and measures is from Table 2 and Appendix A in Geisler, “An Integrated Cost-Performance Model of Research and Development Evaluation,” Omega, 23(3), 1995, 281-294.

1. Economic Benefits and Costs1.1 Improved gross national product.1.2 Improvements in the balance of trade and balance of payments.1.3 Improved GDP/capita.1.4 Costs of living in a highly technological economy.

2. Social Benefits and Costs2.1 Improved level of overall satisfaction and happiness of population.2.2 Expanded “middle” or “professional” class.2.3 Costs of living in a technological society.

WORKSHOP

IN

MONTREAL

ILLUSTRATIVE ORGANIZATION-SPECIFIC INDICATORS AND MEASURES OF THE OUTPUTS FROM S&T

1. Level of Technical Expertise

1.1 Ratio of doctorate holders to scientific workforce.

1.2 Relative experience of scientists and engineers (total years of technical work).

2. Attractiveness of the R&D Organization

2.1 Number of candidates applying for scientific position, per position.

2.2 Age profile of scientists and engineers.

2.3 Judgment of quality of the organization by peers.

WORKSHOP

IN

MONTREAL

II. INTERMEDIATE OUTPUTS

1. Level of Investment in Exploitation of R&D1.1 Funds allocated to technology commercialization by the

focal organization (e.g., company, government agency).1.2 Number of personnel from non-R&D units working with

R&D units (per scientists & engineers).2. Level of Importance of R&D Outcomes

2.1 Role of new products/services in the organization’s success and survival (as perceived by senior managers of the organization).2.2 Perceived success of the transfer process and implementation/absorption of R&D outputs in the organization (perceived by senior managers of the organization).

3. Climate and Leadership3.1 Degree of support that senior management gives to R&D generation, adoption, and transfer.3.2 Overall “climate” in the organization: favorable or unfavorable to R&D (perceived by workers and managers).

WORKSHOP

IN

MONTREAL

III. PRE-ULTIMATE OUTPUTS

1. Investments in Adoption of R&D1.1 Funds allocated to the total adoption, adaptation, and utilization of R&D intermediate outputs by the focal organizations (as ratio of total budgets, and over time).1.2 Ratio of products, services, and processes impacted by R&D outputs that are adopted, implemented, and utilized by the organization.

2. Structure of the Industry2.1 Structural variables such as size, centralization, and vertical integration.2.2 Traditional perception of S&T in the industry: How important is R&D to the progress and survival of the industry?

3. Strategy and Life Cycle3.1 Role of R&D in strategic management of the industry and

sector.3.2 Impact of R&D on the industry in its stage of life cycle.

WORKSHOP

IN

MONTREAL

IV. ULTIMATE OUTPUTS

1. Role and Importance of R&D

1.1 Role of R&D in the economy, as perceived by the population.

1.2 Role of R&D in social progress, as perceived by the population.

1.3 Importance of R&D as a political issue.

2. S&T Level of Population

2.1 Levels of R&D literacy in population.

2.2 Acceptance of the importance of investments in R&D by the population.

Source for some of the indicators and measures if from Table 2 and Appendix A in Geisler, “An Integrated Cost-Performance Model of Research and

Development Evaluation,” Omega, 23(3), 1995, 281-294.

WORKSHOP

IN

MONTREAL

COMPUTATION OF LOADING OUTPUTS INDICATORS

LO I IV ws i isi

n

1

IV d wi ia iaa

n i

1

( )

s = stage pf [process/outputs )1=4_n = number pf ;leading output indicatorsi = each indicatorw = weight of ith indicatorIV = index value of indicator,

so that

wia = weight of the ath measure of indicator Idia = each component indicatorn(i) = number of measures of ith indicatoria = value of ath measure of indicator i

WORKSHOP

IN

MONTREAL

KEYOUTPUT INDICATOR

W^

KOI = LOIsWs^

= normalized weights for each LOI

(This is the integration of all four stages, from immediate to ultimate outputs.)

WORKSHOP

IN

MONTREAL

MEASURING IMPACTS BY STAGE

(alpha) = KOI for LOI1

(immediate outputs)

These are outcomes from the public laboratory.

(beta) = KOI for LOI1-2

(immediate + intermediate outputs)

These are outcomes from the public laboratory & inputs to parent agency, industry, and society.

WORKSHOP

IN

MONTREAL

(gamma) = KOI for LOT1-3

(immediate, intermediate, andpreultimate outputs)

These are outcomes from public laboratoriesand impacts on parent agency, industry, and society.

(omega) = KOI for LOT1-4

(immediate ultimate outputs)

These are overall impacts on society’s goals andobjectives.

WORKSHOP

IN

MONTREAL

ADVANTAGES OF PROCESS-STAGES METHOD

- format allows for choice of “upstream” or ‘downstream emphasis in the measurement of outputs.

• This format allows for comparison of outputs and impacts between “middle” and “downstream” for the same public laboratory.

• This format allows the evaluator (including management at laboratory or parent agency, or constituents) to assign different weights to each stage and to the “upstream” or “downstream” portion of the process, thus tailoring the evaluation to strategic objectives.

WORKSHOP

IN

MONTREAL

Illustrative Development of Indexes of Leading Output Indicators Illustrative Development of Indexes of Leading Output Indicators for Immediate and Intermediate Outputs for Two Public for Immediate and Intermediate Outputs for Two Public

LaboratoriesLaboratories

IndicatorIndicator MeasuresMeasures Time PeriodTime Period WeightWeight BenchmarkBenchmark

1.1. Scientific & Scientific & technical technical outputsoutputs

2.2. Hardware/ Hardware/ software/other software/other originsorigins

20 publications in refereed 20 publications in refereed journalsjournals

15 patent applications15 patent applications

12 new/improved test 12 new/improved test methods methods

suggested 3 standard suggested 3 standard developeddeveloped

9/02-9/039/02-9/03

9/02-9/039/02-9/03

9/02-9/039/02-9/03

9/02-9/039/02-9/03

3535

6565

6060

4040

3030

1010

55

33

Laboratory ACore Indicators

(1) Immediate Outputs


1.1. Technical Technical expertiseexpertise

2.2. Attractiveness Attractiveness of of organizationorganization

10% doctorate holders10% doctorate holders

6 years average 6 years average experience of S & Esexperience of S & Es

3 candidates on average3 candidates on average

applying for S & E applying for S & E positionspositions

[200 miles] the proximity [200 miles] the proximity to nearest universityto nearest university

9/039/03

9/039/03

9/02-9/039/02-9/03

9/039/03

5050

5050

7070

3030

30%30%

5 yr5 yr

5 candidates5 candidates

20 miles20 miles

Laboratory AOrganization-Specific Indicators

From the formula in equation (3):

IV1 = 16.75 (Benchmark = 17.00)IV2 = 8.40 (Benchmark = 5.00)IV3 = 8.00 (Benchmark = 17.50)IV4 = 0.30 (Benchmark = 0.90)

Weights assigned to IV1:

IV1 = 0.4IV2 = 0.2IV3 = 0.2IV4 = 0.2

Applying equation (4): LOI(A)1 = 10.04 (Benchmark = 11.48)

Laboratory BCore Indicators


1.1. Scientific & Scientific & technical technical outputsoutputs

2.2. Hardware/ Hardware/ software/other software/other originsorigins

43 publications in refereed 43 publications in refereed journalsjournals

7 patent applications7 patent applications

2 new/improved test 2 new/improved test methods methods

about 20 about 20 useableuseable ideas/ ideas/

concepts transferred concepts transferred downstreamdownstream

9/02-9/039/02-9/03

9/02-9/039/02-9/03

9/02-9/039/02-9/03

9/02-9/039/02-9/03

7575

2525

1010

9090

3030

55

00

2020


1.1. Technical Technical expertiseexpertise

2.2. Attractiveness Attractiveness of of organizationorganization

30% doctorate holders30% doctorate holders

8 years average 8 years average experience of S & Esexperience of S & Es

6 candidates on average6 candidates on average

applying for S & E applying for S & E positionspositions

[10 miles] the proximity to [10 miles] the proximity to nearest universitynearest university

9/039/03

9/039/03

9/02-9/039/02-9/03

9/039/03

0000

1010

2020

8080

30%30%

5 yr5 yr

5 candidates5 candidates

20 miles20 miles

Laboratory BOrganization-Specific Indicators

From the formula in equation (3):

IV1 = 23.75 (Benchmark = 23.75)IV2 = 18.00 (Benchmark = 18.00)IV3 = 27.50 (Benchmark = 27.50)IV4 = 2.40 (Benchmark = 2.40)

Weights assigned to IV1:

IV1 = 0.6IV2 = 0.1IV3 = 0.2IV4 = 0.1

Applying equation (4): LOI(B)1 = 28.02 (Benchmark = 21.75)

Laboratory A


1.1. Scientific Scientific technical impact technical impact on direct user of on direct user of R&D outcomesR&D outcomes

1.1. Economic Economic impacts on direct impacts on direct users of R&D users of R&D outcomesoutcomes

2 new/improved products2 new/improved productsimproved testsimproved tests

$10K actual cost reduction $10K actual cost reduction in manufacture of a productin manufacture of a product

$20K actual improvement $20K actual improvement in a technique for testing in a technique for testing materialsmaterials

9/039/039/039/03

9/02-9/039/02-9/03

9/02-9/039/02-9/03

60604040

9090

1-1-

2222

NANA

NANA

(2) Intermediate Outputs

By applying equation (3):IV1 = 2.4 (Benchmark = 2.0)IV2 = 11.10 (Benchmark = NA)

Weights assigned to IV1:IV1 = 0.3IV2 = 0.7

LOI(A)2 = 8.42 (Benchmark = N/A)


1.1. Scientific Scientific technical technical impacts on impacts on direct user of direct user of R&D outcomesR&D outcomes

2.2. Economic Economic impacts on impacts on direct users of direct users of R&D outcomesR&D outcomes

(1)(1) new/improved productsnew/improved products(2)(2) improved testsimproved tests(3)(3) actual cost reductionsactual cost reductions

$100K actual improvement in $100K actual improvement in a technique for materials a technique for materials handling processhandling process

9/039/039/039/03

9/02-9/039/02-9/03

9/039/03

808020205050

5050

NANANANANANA

NANA

Laboratory B

By applying equation (4): LOI(B)2 = NA

By applying equation (3):IV1 = 0 IV2 = 50

(weight assigned = 0.8)(weight assigned = 0.2)

(3) KOI for Laboratory A

(4) KOI for Laboratory B

Weights assigned: LOI(A)1 (immediate outputs) = 0.3 LOI(A)2 (intermediate outputs) = 0.7

From equation (5): KOI(A) = (10.04) 0.3 + (8.42 0.7) = 8.90

Weights assigned: LOI(B)1 (immediate outputs) = 0.8 LOI(B)2 (intermediate outputs) = 0.2

From equation (5): KOI(B) = (28.02) 0.8 + (10.0) 0.2 = 24.41

workshop in montreal

Documents

rd process

public rd

overall process

public sector laboratories

economic systemsinputs

economic benefit

impacts of science

technology transfersuccess