understanding effective analytic collaboration

Analytic Collaboration Study November 2009

Lee Scott Ehrhart, PhD 1

Understanding Effective Analytic Collaboration

Lee Scott Ehrhart, PhDNovember 2009

November 2009 1

UNCLASSIFIED

Overview

• The Intelligence Community (IC) quest for improved analytic collaboration and information sharing often focuses attention on technology support

• In 2008, I was asked to study the social, behavioral, and organizational characteristics of a highly‐productive analytic team

• Study Findings– Examination of team products and processes using qualitative and

quantitative methods provided an objective explanation for the subjective assessments of the cell’s effectiveness

– Collaborative reporting on the intelligence issue both Increased and Diffused with the formation of analytic cell

– Co‐location accelerated tradecraft development

2November 2009



UNCLASSIFIED

Award-Winning Team

• The analytic cell examined in this study was funded and established to improve fused multi‐INT reporting on a high‐priority intelligence issue

• Within six months after standup, the cell dramatically impacted the Community knowledge (scope, characteristics, and processes) and earned multiple analytic collaboration awards– Published first multi‐seal reports

– Set new standards in innovative, end‐to‐end issue reporting

• Senior leadership wanted to understand the reasons behind the team’s success hoping to replicate that performance in other analytic efforts

3November 2009

UNCLASSIFIED

STUDYING ANALYTIC COLLABORATION

Examining the Socio‐Behavioral and Organizational Characteristics of a Highly‐Effective Collaborative Cell

4November 2009 4



UNCLASSIFIED

Study Goals

• Understand the Team’s Success• Glean Best‐Practice Models for Future Teams

– Workflow modeling of analytic and organizational processes

– Identification and categorization of key factors for analytic and organizational success

• Suggest how best to extend success to virtual teams– Team dynamics– Organizational structure, policies, and processes– Technology affordances

5November 2009

UNCLASSIFIED

Method

• Direct Observation – In the cell

– Email monitoring

• Interviews and conversations with analysts and managers – including critical incident reviews

• Statistical analysis of published products– Name, product role, organizational affiliation (agency, division, branch) for

reporting on 11 key targets (Oct 2001 – Aug 2008

– Cross‐boundary collaboration trends among the principal stakeholders working the issue

• Social network analysis of collaboration patterns

6November 2009



UNCLASSIFIED

Team Productivity

7

• Teams without an igniting purpose, lack focus and become friendly “Country Clubs”

– Some senior leaders expressed this fear when team was proposed – but direct supervisors believed the igniting purpose was sufficient to ensure performance

• “Silos” exist where teams erect barriers or are forced to exist in a closed environment

– “Silos” represent more than technology or access barriers – they represent a fundamental mindset about “lanes in the road” and work assignments

• Limited cooperation or goodwill within the team “Big Freeze”

CollaborativeMindset

IgnitingPurpose

BoundarySpanning

Big Freeze

NormalBusiness

Zone

Creative, Highly Productive

“Hot Zone”

Low Productivity “Cold Zone”

Adapted from Gratton, 2007

Productive Hot Spots occur where teams share an “igniting” purpose, a cooperative mindset, and willingness to span organizational boundaries(Gratton, 2007; Lin et al, 2008; Lurey et al, 2001)

November 2009

UNCLASSIFIED

Collaboration Network – Before Team

8

Before Team

• A few senior analysts began an informal collaboration spanning several sources, functional organizations, and agencies

• Individual members connected through introduction and common work assignments (e.g., tours of duty in branches or special locations)

• Earliest boundary spanners seeded collaboration at each assignment

• Initial inter‐agency relationships were developed over a long period (~ 20 yrs) without active support

• Shared interests lead to establishing an analytic working group spanning multiple agencies – the primary inter‐agency bridge before co‐location

• Who collaborates with whom• Which analysts are best connected within the organization• Which analysts are boundary-spanners

November 2009



UNCLASSIFIED

Collaboration Network – With Team

9

With Team

• Original network expanded to include more analysts –internal and external to agency – and expand the representation of the problem

• Cross‐cutting team from several agencies collocated to foster real‐time collaboration

• Original boundary spanners expanded their boundaries; new boundary spanners developed –creating a collaborative “hot spot”

• Identification of the unit as a collaborative, information sharing team increased team cohesion and enhanced the relationships with external agencies

• Some diminution of inter‐branch collaboration as most of the branch analysts working team issues are assigned to unit

• Who collaborates with whom• Which analysts are best connected within the organization• Which analysts are boundary-spanners

November 2009

UNCLASSIFIED

Collaboration Network

10

External Agencies

• High degree of cohesion and collaboration among core team ANDprimary external organizations

• Addition of new team members and/or components extends the network and adds new external collaborators

• Full network is not engaged in every effort, but the best set of participants can be rapidly assembled to meet a challenge

Intra/Inter-Organizational Collaboration

November 2009



UNCLASSIFIED

Distribution of Collaboration Levels

11

Level 15%

Level 28%

Level 363%

Level 413%

Level 511%

Average Before Team

Level 12%

Level 21%

Level 330%

Level 428%

Level 539%

Average with Team

Oct 2001 – Mar 2007• 87% of reports involve collaboration outside of agency branch boundaries

• Most collaboration occurring across branches within agency (Level 3 = 63%)

• Less than 25% of the reporting involved external agency collaboration (24%)

• Coordination with external agencies (13%)• Collaborative reporting with external agencies (11%)

Apr 2007 – Aug 2008• 98% of reports involve collaboration outside of branch boundaries

• Majority of the reporting involved external agency collaboration (67%)

• Coordination with external agencies (Level 4 = 28%)

• Collaborative reporting with external agencies (Level 5 = 39%)

Single author / Single branch

Multiple authors / Singlebranch

Multiple authors / Multiple Branches

Coordination with 1 or more external agencies

Contributions from 1 or more external agencies

November 2009

UNCLASSIFIED

Increased Multi-Agency Collaboration

12

Oct 2001 – Mar 2007• <25% of the reports involve another agency (24%)

Apr 2007 – Aug 2008• 63% of the reports involve collaboration with one or more agencies

76% Host Agency Only 37%

9% Agency + 1 Agency 21%

7% Agency + 2 Agencies 11%




76%

9%

7% 6%

2% 0%

Before Team

37%

21%

11%

13%

15%3%

With Team

November 2009



UNCLASSIFIED

Cross-Boundary Collaboration Before Team

13

5 analysts are authoring most of the reports that cross branch and/or agency boundaries

7 analysts have authored reports involving external coordination (Level 4) or collaboration (Level 5)

Only 1 analyst has written more than 2 multi-agency collaboration reports (Level 5)

Level 3 – Multiple authors / Multiple branches within unit host agency

Level 4 – Coordination with one or more external agencies

Level 5 – Collaboration with one or more external agencies

0

5

10

15

20

25

30

35

40

Level 3Level 4

Level 5

8

9

5

2

21

3

12

9

33

32

2

2 32

Nu

mb

er o

f R

epo

rts

Before Team

November 2009

UNCLASSIFIED

Cross-Boundary Collaboration with Team

14

All 15 analysts – including junior analysts – authored reports that engaged an external agency for coordination (Level 4) or collaboration (Level 5)

5 analysts produced more than 2 multi-agency collaboration reports (Level 5)

Level 3 – Multiple authors / Multiple branches within host agency

Level 4 – Coordination with one or more external agencies

Level 5 – Collaboration with one or more external agencies

Nu

mb

er o

f R

epo

rts

0

5

10

15

20

25

30

35

40

Level3

Level4

Level5

10

12

5

9

22

3

1519

3

6

36

7

6

3

4

3 3

With Team

November 2009



UNCLASSIFIED

Key Findings

• Creation of a Collaborative Analytic Cell Increased and Diffused Collaborative Reporting Increased number and percentage of products from cross‐boundary collaboration Increased number of agencies and organizations participating in collaborative

reporting Increased number of analysts authoring and contributing to multi‐INT, multi‐agency

reports

• Reporting reflects broad and deep intra‐agency and inter‐agency collaboration– Intra‐agency (crossing multiple branches and multiple offices) – 32 branches across

10 divisions within the host agency cited as author, co‐author, contributor, or coordinator

– Inter‐agency (co‐author, contributor, or coordinator) – 21 branches across 6 external agencies

– Increase in collaborative reporting by junior analysts

15November 2009

UNCLASSIFIED

Collaboration Network Lessons Learned

• Boundary spanners provide the “anchor stores” for a successful collaborative enterprise

• Boundary spanning analysts are not that difficult to identify– Most analysts know who the boundary spanners are within their working

group

– Existing and potential boundary spanners can be identified by patterns of reporting

• Analytic working groups and issue conferences build inter‐agency contacts and foster boundary spanning behavior

• Boundary spanning can be cultivated given– Culture of collaboration

– Focused issue

– Access to relevant information resources, and

– Management support

16November 2009



UNCLASSIFIED

Increasing Team Effectiveness

• Information Sharing Practices – encouraging team members to share

information and build a shared sense of competencies and trust

• Collaboration Support – using technology to bring distributed team

members closer to the team

• Work Flexibility – enabling team members to work at different paces with

sufficient integration to pull efforts together

• Progress Feedback – providing feedback to team members to coordinate

their activities in support of the team objectives

• Conflict Resolution Practices – encourage surfacing and working

through issues; valuing different opinions and alternative interpretations

• Time for Reflection – establishing a work pace that supports time pressure

demands interspersed with times for reflection and purposeful conversation

17November 2009

UNCLASSIFIED

EXTENDING COLLABORATION IN VIRTUALENVIRONMENTS

Applying Lessons Learned to Virtual and Hybrid Collaborative Communities

18November 2009 18



UNCLASSIFIED

Team Effectiveness

• Issues that affect virtual teams are similar to those that affect co‐located or hybrid teams

• Team effectiveness is most impacted by team leadership– Establishing positive team processes– Developing supportive team member relations– Creating team‐based reward systems– Selecting only those team members who are qualified to do the

work

• Teams whose members are aware of the knowledge and expertise of other teams members perform better than teams whose members do not possess such knowledge

November 2009 19

UNCLASSIFIED

Supporting Virtual Collaboration

• Develop means to find collaborators and build new contacts– Most analysts know how to find other analysts who are working issues similar

to theirs via published reporting, blogs, Intellipedia, A‐Space, etc. – the support is growing daily

– Short‐term co‐location helps develop social relationships and trust that carries over into other assignments

– Regular face‐to‐face or VTC contact helps to extend 3rd party introductions and reinforce the social network

• Create incentives for sharing information across boundaries

• Support the loose coupling of strong small cells to other cells to form a hybrid network of co‐located and virtual teams (Granovetter, 1973, 1983)

20November 2009



UNCLASSIFIED

Organizational Networks

• An organization can be modeled and characterized as a set of interlocked networks connecting entities– People

– Knowledge

– Resources

– Tasks/Projects

• Carley and Reminga (2004) represent these interlocked relationships using a meta‐matrix conceptual framework

• This study applied Carley’s basic framework to examine the differences and similarities between co‐located collaborative networks and virtual/hybrid collaborative networks.

21November 2009

UNCLASSIFIED

Organizational Network Relationships

People Knowledge Resources Tasks / Projects

People

Collaboration / Coordination NetworkWho talks to, works with, and reports to whom

Knowledge NetworkWho knows what, has what expertise or skills

Resource NetworkWho has access to or can use which resource

Assignment NetworkWho is assigned to which task or project, who does what

Knowledge

Information NetworkConnections among types of knowledge,mental models

Resource Usage RqmtsWhat type of knowledge is needed to use that resource

Knowledge RqmtsWhat type of knowledge is needed for that task or project

Resources

Inter‐operability & Co‐usage RqmtsConnections among resources, substitutions

Resource RqmtsWhat type of resources are needed for that task or project

Tasks / Projects

Precedence & DependenciesWhich tasks are related

Adapted from K. M. Carley & J. Reminga, 2004.

22November 2009 22



UNCLASSIFIED

Best Practices for Virtual Teams (1)Issue Co‐located Teams Virtual / Hybrid Teams

Collaboration / Coordination NetworkWho talks to, works with, and reports to whom

• Develop social relationships necessary for team effectiveness through face‐to‐face interaction

• Team cohesion is based on shared objectives and trust built through personal interaction

• Internal team leadership has benefit of face‐to‐face contact

• Need avenues for personal contact to build trust and encourage collaboration

• Local team meetings (e.g., inter‐agency working groups)

• Regular video or audio teleconferences• Internal team leadership is critical and must be exercised in the virtual environment

Knowledge NetworkWho knows what, has what expertise or skills

• Leverage physical proximity to engage in issue discussions informally

• Ability to pose small questions to the team for quick answers

• Access to special expertise may be limited by location

• Lack the benefits of conversational proximity

• Added connectivity – including audio chat capabilities to simulate co‐location

• Access to experts can be extended virtually to all team sites – but requires a request process and knowledge of available experts

See also: Straus et al, 2008; Weick & Sutcliffe, 2005.

November 2009 23

UNCLASSIFIED


ResourceNetworkWho has access to or can use which resource

• Access to resources can be resolved locally – but may limited to locally available resources

• Access to resources must be resolved for team members, regardless of physical location

• Common share folders• Well‐understood procedures for conveying information from non‐shared resources

Assignment NetworkWho is assigned to which task or project, who does what

• Co‐location results in tightly coupled structure with looser definition of roles

• Proximity permits agile work assignment, based on need and available team members

• Distributed teams support a loosely coupled organization, but require clearly defined roles and expectations

• Agility and responsiveness must be cultivated as part of the team process (ex., common team email)

November 2009 24



UNCLASSIFIED


Information NetworkConnections among types of knowledge,mental models

• Connections between issues or problem components surface through conversation

• Team members engaged in shared sensemaking both synchronously and asynchronously

• Shared mental models (transactive memory) help to focus information sharing among team members

• Connections between issues require explicit discussion• Working group discussions – virtual or face‐to‐face

• Support for synchronous and asynchronous discussion – e.g., email, group forums, electronic messaging

• Find an Expert web resources aid in instantiating and promulgating the group memory• Helps identify the best team member to reach for a specific issue or question

• Example: Team Intellipedia site

See also: Weick & Sutcliffe, 2005.

November 2009 25

UNCLASSIFIED


Resource Usage RqmtsWhat type of knowledge is needed to use that resource

• Use of unique resources, such as visualization tools or special sources, may

be limited by expertise and training

• Co‐located teams may or may not have

special resources available or training to

use them and may have to compete with

other teams

• Virtual teams are equally resource‐limited

for special skill categories in some

locations

• Virtual connection permits the sharing of

resource expertise that is not co‐located

with team

Knowledge RqmtsWhat type of knowledge is needed for the task or project

• Productivity is enhanced when all the essential knowledge of the relevant

issues is contained within the team or

readily available through existing

contacts

• Where a team lacks specific expertise, it

may take longer to develop needed

knowledge – or never adequately address

the issue

• Virtual teams can be built rapidly to meet

the task with knowledge and enhanced

with new virtual team members to

encourage innovative thinking and

comprehensively address an issue

November 2009 26



UNCLASSIFIED


Inter‐operability& Co‐usage RqmtsConnections among resources, substitutions

• Use of resources may be limited by training and/or access to information or tools

• Co‐located teams may have to share access to limited resources (e.g., single workstation connectivity)

• Resource access may create a bottleneck or block to collaboration (e.g., team member cannot access needed resource from a particular location)

• Workarounds can be burdensome

• Virtual connectivity to resources can eliminate the problem faced by team member with access, but no connectivity

• Virtual teams need the ability to establish their standard working environment regardless of work location

November 2009 27

UNCLASSIFIED


Resource RqmtsWhat type of resources are needed for that task or project

• Resource requirements for task are similar in both team configurations – access to resources is the primary difference

• Resource requirements for task are similar in both team configurations –access to resources is the primary difference

Precedence & DependenciesWhich tasks are related

• Organizational boundaries can limit the ability of analysts to cooperatively trace the end‐to‐end process or recognize the inter‐relationship of issues

• Virtual collaboration may permit the inclusion of additional view points or expertise, leading to more innovative understanding of the issue and its components

November 2009 28



UNCLASSIFIED

Summary

• Examination of products and processes using qualitative and quantitative methods provides objective explanations for a productive “Hot Spot”– Who collaborates across organizational boundaries

– Who leads collaborative activities

– Dimensions and diffusion patterns for informal collaborative networks

• Significant benefits from integrating issue versus regional or functional organization– Increased percentage of products from cross‐boundary collaboration –

both intra‐ and inter‐agency

– Increased diffusion of analytic collaboration across Community on the intelligence issue

– Accelerated tradecraft development among junior analysts

29November 2009

UNCLASSIFIED

BACKUP

Other Observations

Selected References

30November 2009 30



UNCLASSIFIED

Observations (1)

• Before Team Established– Initial collaboration began as a semi‐virtual association – Built from direct working relationships and personal

introductions– Little direct support and tacit approval from management as

long as collaboration contributed to the production goals

• Effort to Create Collaborative Team– Required persistence on the part of the primary founders– Proposal was not universally well‐received by senior leadership– Innovation Grant and enthusiasm of key managers helped gain

the support necessary to establish the unit in April 2007

31November 2009 31

UNCLASSIFIED

Observations (2)

• Collaborative unit stood up in April 2007 (2Q 2007)– Level 3 (intra‐agency collaboration) increased immediately– Level 4 (inter‐agency coordination) doubled by 3Q 2007– Level 5 (inter‐agency collaboration) more than tripled by 4Q

2007 – approximately 9 months after standup

• Early challenges lead to significant successes and 4Q 2007 was a productivity high point based on numbers only– Volume of reporting varies with level of relevant activity

• The more relevant indicator is the level of innovation in the reporting– Publication of the first multi‐seal reports– New standards set in end‐to‐end issue reporting

32November 2009 32



UNCLASSIFIED

Observations (3)• Management lessons learned were addressed in detail in

a joint brief by host agency managers– Confirmed the need for mid‐level managers to protect and nurture a

highly productive team spanning multiple offices and branches– Highlighted the value of applications of basic technology – email

distribution lists – to aid in coordination and reduce management meddling

– Discussed the problems of extending the virtual collaboration across multiple network channels and varying access to information resources

• Team continues today as a semi‐virtual collaboration– Core boundary‐spanning agency team is collocated, but

accommodates other agencies with onsite access to their network resources

– Collaboration across network boundaries and with distributed participants is still accomplished largely through email and phone calls

– Collaborative production of reports continues to require elaborate workarounds

33November 2009 33

UNCLASSIFIED

Understanding Collaborative Analysis• The Intelligence Community (IC) emphasis on collaboration

and information sharing often focuses attention on technology

• Premise: Successful collaboration is best understood by focusing on team social, behavioral, and organizational factors

• Study Findings: – Examination of products and processes using qualitative and

quantitative methods provided an objective explanation for the subjective assessments of the cell’s effectiveness

– Collaborative reporting on the intelligence issue both Increasedand Diffused after the formation of a issue‐focused cell

– Roots of collaboration traced back many years before the creation of the co‐located team

34November 2009



UNCLASSIFIED

Research on Collaborative Teams• Hot Spots occur where teams share an “igniting” purpose, a cooperative

mindset, and willingness to span organizational boundaries(Gratton, 2007; Lin et al, 2008; Lurey et al, 2001)

• Opportunities exist to nurture hot spots – Focus on a meaningful team goal– Foster the collaborative network– Remove barriers to cooperation

• Over time, teams develop transactive memory and shared mental models that permit individual members to serve as external memory aids to each other (Wegner, 1986)

• Virtual teams are first and foremost teams (Davenport, 2005; Lurey et al, 2001; Walvoord, 2008)

– Must have a shared purpose – joint goals – to foster the need to collaborate– Joint goals require team members to rely on each other– Trust is a critical component of effective teamwork

November 2009 35

UNCLASSIFIED

Selected References• Carley KM and Reminga J. ORA: Organization Risk Analyzer. Technical Report: CMU‐ISRI‐04‐106. Center for

Computational Analysis of Social and Organizational Systems (CASOS), Carnegie Mellon University, Jan 2004.• Davenport TH. Thinking for a Living; How to Get Better Performance and Results from Knowledge Workers.

Harvard Business School Press, 2005.• Granovetter, M. The strength of weak ties. Am. J. of Sociology, 78(6) May 1973: 1360‐1380.• __________. The strength of weak ties: A network theory revisited. Sociological Theory, 1, 1983: 201‐233.• Gratton, L. Hot Spots; Why Some Teams, Workplaces, and Organizations Buzz with Energy – and Others

Don’t. Berrett‐Koehler, 2007.• Kanawattanachai P & Yoo Y. Dynamic nature of trust in virtual teams. Sprouts: Working Papers on

Information Environments, Systems and Organizations, 2(Spring 2002), 42‐58.• Lin C, Standing C, and Liu Y‐C. A model to develop effective virtual teams. Decision Support Systems,

45(2008): 1031‐1045.• Lurey JS and Raisinghani MS. An empirical study of best practices in virtual teams. Information &

Management, 38(2001): 523‐544.• Straus SG, Parker AM, Bruce JB, and Dembosky JW. The Group Matters; a Review of the Effects of Group

Interaction on Processes and Outcomes in Analytic Teams. Project Memorandum: PM‐2392‐1‐USCA. RAND, National Security Research Division, Washington, DC: Feb 2008

• Walvoord AAG, Redden ER, Elliott LR, and Coovert MD. Empowering followers in virtual teams: Guiding principles from theory and practice. Computers in Human Behavior, 24 (2008): 1884‐1906.

• Wegner DM. Transactive memory: A contemporary analysis of the group mind. In Mullen B & Goethals GR (Eds.), Theories of Group Behavior. Springer‐Verlag, 1987.

• Weick KE and Sutcliffe KM. Organizing and the process of sensemaking. Organization Science, 16(4), July‐Aug 2005: 409‐421.

36November 2009 36

understanding effective analytic collaboration

Documents