a sociotechnical systems study of virtual r&d organizations

A Sociotechnical Systems Study of Virtual R&D Organiza;ons

Co-Chairs Ronald E. Purser & Ramkrishnan V. Tenkasi

Panelists: Doug Austrom Betty Barrett Bert Painter

Pamela Posey

Discussant: William A. Pasmore

Sponsors: ODC, OCIS, TIM

Supported by NSF-VOSS award #0943237

Central Research Ques;on How do virtual modes of communica3on influence the quality of delibera3ons at various stages of the R&D process?

-‐  STS theore3cal lens -‐  Compara3ve case study -‐  Grounded theory building

Session Overview •  Introduc3on and Framing: 1:15-‐1:30

Panel Presenta3ons •  Large Video Game Inc 1:30-‐1:40 •  Alzheimer’s Disease center 1:40-‐1:50

•  CALTECH Micro & Nano-‐photonics 1:50-‐2:00 •  Summarizing across the cases 2:00-‐2:15

•  Discussant comments 2:15-‐2:30

•  Q& A 2:30-‐2:45

Central Research Ques;on How do virtual modes of communica3on influence the quality of delibera3ons at various stages of the R&D process?

-‐  STS framework -‐  Delibera3ons -‐  Stages of the R&D

Evolu3on of Socio-‐Technical Systems

STS v1.0 Rou3ne work in a single organiza3on – e.g., coal mines, factories, oil refineries

• Work groups with pooled iden3ty •  Unitary conversion process •  Linear conversion sequence

STS v2.0 Non-‐rou3ne knowledge work in single organiza3ons – e.g., white collar office work, professional services firm, NPD and R&D

•  Individual performers, specialized exper3se

• Mul3ple, concurrent conversion processes

•  Nonlinear conversion flow

STS v3.0 Virtual, non-‐rou3ne work – e.g., R&D consor3a, complex supply chains

•  Individual performers and work groups distributed across mul3ple loca3ons and/or organiza3ons

•  ICT enabled • Mul3ple, concurrent independent and interdependent conversion processes

•  Nonlinear conversion flows

*

Delibera;ons: Defini;on and Elements

Pava, 1983

•  Delibera;ons are sense making exchanges in which people engage with themselves or others to reduce the uncertainty of a problema9c issue. -‐-‐i.e. an issue subject to two or more interpreta9ons

•  The salient elements of a delibera3on include the … •  Topics or problema3c issues facing the social en3ty about which people reflect and communicate

•  Forums in which they occur which may structured, semi-‐structured, or unstructured or ad hoc

•  Par;cipants both those who are currently involved and those who ideally should be involved in the delibera3on.

A Six Stage Con;nuum of the R&D Process

Sampling of Virtual R&D Projects across Sites and R&D Stages

Type of Research Development Program/Project

Research Site R1 Pure

R2 Applied

D1 Exploratory

D2 Advanced

D3 Start-up

D4 Scale-up

Large Video Game Inc.

X X

NIA Alzheimer’s Centers

X X X

Caltech Micro & Nano-Photonics Research Group

X X

Large Video Game, Inc.

Case Study

Major support and funding for this project has been provided by the Na;onal Science Founda;on.

NSF Award #0943237

LVG Inc. Core Team

Florida

QA Tes;ng

Southern USA

Engineering and Web Development

USA

LVG Inc.

Canada

Artwork (Philiphines, India, China)

Virtual Organiza;on for LVG Inc. Game 2012

Product Development

Systems Engineering

China

NSF Award #0943237

Systems View of Game Development

NSF Award #0943237

Core Game Team

NSF Award #0943237

QA

Art

Engineering Production

Design

Org Structure

NSF Award #0943237

Producer & Director

Art Director Lead Designer Technical Director

Engine/System Coders Gameplay Coders Network Coders

Hardware Specialists Audio Specialists

Rendering Specialists

Gameplay Designers SP Level Designers Scripting Specialists Cinematic Specialists MP Level Designers

Concept Artists Character Artists

Environment Artists 3D Modelers

Texture Artists Technical Artists

Animators

Audio Director

Assistant Producer Writers

Sound Engineers Music Composer Recording Studio

Voice Actors Localization Studio

AGILE -‐ SCRUM •  Rapid prototyping process + mul3ple itera3ons to feature comple3on

•  Episodic surveys of “POD” members, assessing their virtual interac3ons with vendors aler every SPRINT cycle (3 week cycles, 13 total SPRINT cycles, we entered at SPRINT 6).

•  AGILE has significant implica3ons for virtual organiza3ons insofar as expecta3ons may change during the life of a contract, during various “Sprints”

NSF Award #0943237

SCRUM BOARD

Feature to develop Tasks in queue

Tasks in progress Tasks complete Feature complete

NSF Award #0943237

déjà vu

•  Souder’s NSF research – “Task-‐dominant approach to R&D” (Souder, 1987)

•  “Rugby vs. Relay-‐race” project management

•  How and to what extent do various vendors synchronize and entrain with itera3ve nature of AGILE/SCRUM at the host organiza3on?

NSF Award #0943237

NSF Award #0943237

Data Collec3on to Date •  Ini3al Scoping Phone Interviews with Various General and

Project Managers (Spring/Summer 2010) •  On-‐site interviews (14) with “Franchise” team (Sept. 2010)

–  (General Manager, Project & Program Managers, Ar3sts, Solware Engineers, Crea3ve Director, Audio Engineers, Computer Graphics, Outsourcing Managers, Quality Assurance)

•  Milestone telephone interviews with Project Managers, Ar3sts, Engineers and Quality Assurance (Jan. 2011)

•  Episodic electronic surveys at the end of each “Sprint” cycle, and at the end of “Alpha” (@ 3 weeks, 10/10 – 05/11)

•  Electronic surveys of key vendors (May-‐June 2011)

•  Follow-‐up phone interviews with Project Managers, Ar3sts, Engineers and Quality Assurance (Aug. 2011)

NSF Award #0943237

Uniqueness of Producing Art in Virtual Organiza;ons

•  Art studios produce art assets, not games – Large knowledge gap between making art, and making art that works for games

– Oversea vendors – gap larger •  How is tacit knowledge obtained in virtual exchanges? – Some vendors more sophis3cated than others

•  Ar3s3c and aesthe3c judgment is part of the socio-‐technical system

•  Rich, mul3-‐media documenta3on is key

NSF Award #0943237

Socio-‐Technical Systems of Virtual Art Produc;on: Emergent Themes

–  Rou;ne vs. Nonrou;ne Work. Art assets vary in terms of “Likeness.” The more authen3c and life-‐like the art asset, the less rou3ne, but this actuality reduces “ar3s3c license.”

–  Unit Opera;ons: How to assign and parcel out pieces of work? Too litle turns the vendor into a “monkey,” too large is overwhelming.

–  Expecta;on se`ng is key to joint op3miza3on –  Variances take the form of revisions/ reitera3ons and delays in delivery

– Managing front-‐end delibera;ons is key to seung flexible and realis3c contracts

NSF Award #0943237

Scoping Vendors

Vendor Selection

Defining & Estimating Project Work

Pre-test & Trial Run

Key Delibera;ons in Virtual Art Produc;on

Detailed Documentation & Requirements

Art Assets Deliveries

Initial Testing of Assets

Debugging Re-negotiating Project Work

Contract Negotiation

Critiques

Critiques

NSF Award #0943237

Reducing Equivocality in Art Delibera;ons

•  Contrary to tradi3onal studies on virtual teams that strive for 3ght coupling, ar3s3c produc3on relies more on reducing equivocality of front-‐end delibera3ons by providing well defined documenta3on, visual targets, and reference materials

–  “I write my emails out properly (addressing the person, wri3ng organized paragraphs, and then using closing remarks with a signature). I started no3cing that a lot of that properness started to fade as the sprint carried on. We are star3ng to make the transi3on into alpha and the workload has been plenty, so I thought those emails were a great reflec3on of it. (just something funny I no3ced)” 2/23/11

•  Key delibera3ons in virtual art produc3on involve gaining shared agreement on “standardiza3on of outputs” (Mintzberg, 1993; Ramesh & Dennis, 2002)

•  The use of both “seman3cally and aesthe3cally rich media” (e-‐mails, documenta3on, visual targets, screen shots)

NSF Award #0943237

Virtuality Impacts the Nego;a;on of Art-‐Based Project Work

•  Ambiguity in defining 3me per asset •  Pricing es3mates are olen inaccurate

•  Cost effec3veness is hard to predict –  “Cost nego3a3on. There was a task that required modifying heads. It took us

30min to 1hr to do one head in-‐house, so I tasked them out as 4hr jobs. The outsourcers wanted anywhere from 6-‐16hrs to do, and one agreed to setle at 4hrs per head aler doing 20 of them. Some3mes you win with outsourcing, some3mes it makes no sense to outsource.”

NSF Award #0943237

Management of Risk Virtual Engineering

•  A Key Delibera3on involves managing risk in providing access to source code packages –  Key Variances: Delays in obtaining approvals, incomplete documenta3on, changing expecta3ons/func3onality, mis-‐matches in preferred modes of communica3ons

•  Our biggest challenge with Vendor E has been geung them set up with VPN and the appropriate permissions. There is a lot of red tape that we are having to go through wi3n our company to get them properly set up. It ul3mately took up to 5 weeks to get them fully ready to start work for us. Thus, the project is behind schedule and we are looking at the project coming in 3 weeks late from the original final date. …we have had a ton of email and conference alls trying to figure out the next

steps in geung Egeniq set up.

NSF Award #0943237

Emerging Theory and Prac;ce Ques;ons

•  Can we observe and theorize how to design “high performance” virtual organiza3ons?

•  Can non-‐rou3ne STS principles be applied to virtual art produc3on?

•  Can theories of organiza3onal aesthe3cs help to understand arts-‐based virtual delibera3ons? [Aesthe3cs = sensory knowledge and felt meaning of objects and experiences.]

•  How does aesthe3cs factor into virtual art delibera3ons to create value and performance?

NSF Award #0943237

These Slides are Op3onal

NSF Award #0943237

NSF Award #0943237

LVG Core Team

NSF Award #0943237

Vendors: Factors Impeding Virtual Communica3ons

NSF Award #0943237

NSF Award #0943237

Research Site and Project

• A virtual R&D eco-‐system comprised of 29 NIA-‐funded Alzheimers Disease Centers (ADCs) and the Na3onal Alzheimers Coordina3ng Center Center (NACC)

• Uniform Data Set

Key Peformance Outcomes of of this Alzheimers Research Eco-‐System

Number of enrollees*

23,577 subjects Both demented and non-‐demented

Number of variables

725 variables using up to 18 standardized forms at each visit

Longitudinal data is collected by clinicians, neuropsychologists, and other ADC research personnel each visit -‐-‐ ini3al, annual follow-‐up and milestones such as death or dropout

Number of research projects**

157 research projects have been approved to use the NACC data

•  104 by researchers from the ADCs •  27 by NACC staff •  26 by external researchers

Number of publica;ons using NACC data***

Since 2001 367 publica3ons •  180 NACC-‐funded projects •  53 by NACC staff •  53 NACC-‐ADC collabora3ve studies •  72 by non-‐NACC researchers •  9 from Indirect NACC support

* As of June 2011 ** As of August 2011 *** As of January 2011

Alzheimers Research Eco-‐System and a “Culture of Collabora3on”

•  Culture of collabora3on among Alzheimers researchers has extended and has led to other major collabora3ve research ini3a3ves • Alzheimers Disease Neuro-‐Imaging Ini3a3ve (ADNI) • Genome-‐Wide Associa3on Study (GWAS) • Alzheimers Disease Coopera3ve Study (ADCS) • Dominantly Inherited Alzheimer Network (DIAN) •  Interna3onal collabora3ve ini3a3ve with Worldwide ADNI

Research Process to Date •  Conducted 14 interviews with a representa3ve cross-‐sec3on of key par3cipants in the UDS project including ADC Center Directors, Na3onal Ins3tute of Aging (NIA) ADC Program Director, NACC Director, Clinical Core Directors, and Data Core Directors

•  Determined that the UDS project is D1 in the R&D con3nuum, Exploratory Development work – “They know what; but they didn’t know how to achieve it.”

•  Iden3fied key delibera3ons in the ini3al development, ongoing data collec3on/submission/quality control, and revision of the Uniform Data Set including … •  Iden3fying and choosing the “best fit” data collec3on methods, diagnos3c protocols and instruments, variables for the UDS

•  Achieving agreement on the UDS across the 29 ADCs •  Ensuring adherence with UDS protocols, data quality, and 3meliness of data submission at all 29 centers

•  Revising assessment instruments and protocols in the UDS and achieving agreement on the new instruments

Factors that Contributed to Delibera3on Effec3veness

•  Delibera3on processes itera3vely evolved within the informal network and the formal structure, the NACC

•  Face-‐to-‐face forums twice a year have enabled the ICT-‐enabled (informa3on and communica3on technologies) forums and delibera3ons to be more effec3ve and efficient

•  Relied on high levels of input and par3cipa3on – directly on commitees and sub-‐commitees and via surveys, bulle3n boards, direct feedback to Commitee Chair and Members – to build agreement on the UDS

•  Rou3ne delibera3ons (3mely data submission and data quality are predominantly ICT-‐enabled

Contextual Factors that Contributed to Delibera3on Effec3veness

•  Highly effec3ve and highly collabora3ve virtual socio-‐technical eco-‐sytem has evolved itera3vely in large part because …

•  Leaders in the Alzheimers research community (from the ADCs and NIA) with considerable source credibility (trusted and respected, necessary exper3se) who were commited to the success of the network and a transcendent purpose – i.e., addressing Alzheimers disease

•  NACC has provided the infrastructure for effec3ve and efficient delibera3ons within this virtual STS and serves the func3ons of a referent organiza3on (Trist, 1983) – coordina3on, apprecia3on of emergent issues, and infrastructure support

•  Dense network of rela3onships among par3cipants in the network has also strengthened collabora3on

CALTECH (California Ins;tute of Technology) &

D.A.R.P.A. (Defense Advanced Research Projects Agency)

‘ORCHID’ PROGRAM of Fundamental Research -‐ Op;cal Radia;on Cooling & Hea;ng in Integrated Devices

Theore3cal & Experimental “Opto-‐Mechanics” Research within a Virtual Organiza3on -‐2 project phases, June 2010-‐12 & 2012-‐14:

VOSS study of Phase One-‐-‐Exploratory

-‐Tightly-‐Linked Collabora;on for Design of Experiments & Device Fabrica;on among Laboratories

using 3 Technology plajorms in USA & Europe

-‐ At-‐distance Theore;cal scien;fic Study & Input to work of the Experimental groups

-‐ Major CHALLENGE: Crea3ve Research Design & Knowledge Genera3on

in a complex Virtual se`ng

-‐2 project phases, June 2010-‐12 & 2012-‐14:

VOSS study of Phase One-‐-‐Exploratory

-‐Tightly-‐Linked Collabora;on for Design of Experiments & Device Fabrica;on among Laboratories

using 3 Technology plajorms in USA & Europe

-‐ At-‐distance Theore;cal scien;fic Study & Input to work of the Experimental groups

-‐ Major CHALLENGE: Crea3ve Research Design & Knowledge Genera3on

in a complex Virtual se`ng

Orchid Project Network Core Experimental Groups at CALTECH Physics & Applied Physics Depts.

Quantum Op;cs Group Physics -‐ Caltech

Micro & Nano-‐Photonics Research Group

Applied Physics -‐ Caltech

Meso-‐Op;cs Research Group Applied Physics -‐ Caltech





Photonics & Quantum Measurement Laboratory Federal Inst.of Technology Lausanne, Switzerland

Quantum Op;cs & Nanophysics Group Univ. of Vienna

-‐ Austria

Orchid Project Network Data-‐Sharing & Collabora3on between Caltech & EUROPEAN Laboratories







-‐ Austria

Orchid Project Network THEORETICAL Physicists Study & Input to the work of the ‘Experimentalists’

Theore;cal Physics -‐

McGill Univ. Canada

Op;cal Sciences & Physics -‐

Univ.Arizona







-‐ Austria

Theore;cal Physics -‐

McGill Univ. Canada

Op;cal Sciences & Physics -‐

Univ.Arizona

Orchid Project Network European Graduate Students “Embedded”in Caltech Laboratories

VOSS -‐ Preliminary Finding -‐ Orchid Project -‐ ‘Embedded Researcher(s)’ provide temporary F2F ‘presence’ + ‘transla;on’ assis;ng at-‐distance collaborators

to resolve high ambiguity delibera;ons

-‐NOW, ’Habit’ and sufficient Shared Understandings are established for Skype-‐based collabora;on across groups without ‘embedded’ researcher(s)

-‐Another coincidental forum for F2F delibera;ons -‐ frequent Interna;onal Conferences (>5 per year)

-‐Result: Dynamic Complementary F2F + Technology-‐mediated Interac3ons in ‘hybrid’ Virtual SeHng

(Note: Dixon, K. & Panteli, N. (2010). From Virtual Teams to Virtuality in Teams, Human Rela3ons, 63, 1177-‐1197.)

-‐ ‘Embedded Researcher(s)’ provide temporary F2F ‘presence’ + ‘transla;on’ assis;ng at-‐distance collaborators

to resolve high ambiguity delibera;ons

-‐NOW, ’Habit’ and sufficient Shared Understandings are established for Skype-‐based collabora;on across groups without ‘embedded’ researcher(s)

-‐Another coincidental forum for F2F delibera;ons -‐ frequent Interna;onal Conferences (>5 per year)

-‐Result: Dynamic Complementary F2F + Technology-‐mediated Interac3ons in ‘hybrid’ Virtual SeHng

(Note: Dixon, K. & Panteli, N. (2010). From Virtual Teams to Virtuality in Teams, Human Rela3ons, 63, 1177-‐1197.)

a sociotechnical systems study of virtual r&d organizations

Business

sucient shared

switzerland

nanophysics

cs group physics

caltech micro

caltech photonics

cal sciences

making art