capdem evaluation strategy - defence research …cradpdf.drdc-rddc.gc.ca/pdfs/unc83/p525740.pdf ·...

CapDEM evaluation strategy:Integrating people, process and materiel

CapDEM Evaluation Team

W. RobbinsB. Waruszynski DRDC Ottawa

C. LalancetteM. Lizotte C. NécailleDRDC Valcartier

Defence R&D Canada – ValcartierTechnical Memorandum

DRDC Valcartier TM 2005-164August 2006

CapDEM evaluation strategy: Integrating people, process and materiel

CapDEM Evaluation Team

W. Robbins

B. Waruszynski DRDC Ottawa

C. Lalancette M. Lizotte C. Nécaille DRDC Valcartier

Defence R & D Canada - Valcartier Technical Memorandum DRDC Valcartier TM 2005-164 August 2006

Author

Claire Lalancette

Approved by

Guy Turcotte

Head System of Systems

Approved for release by

Gilles Bérubé

Chief Scientist

© Her Majesty the Queen as represented by the Minister of National Defence, 2006

© Sa majesté la reine, représentée par le ministre de la Défense nationale, 2006

Abstract

The Department of National Defence and the Canadian Forces are currently implementing Capability Based Planning (CBP) as a core element of their overall business process. The Collaborative Capability Definition, Engineering, and Management (CapDEM) Technology Demonstration project aims to define and validate Capability Engineering (CE) as a new means to support CBP by introducing engineering rigour to the realization of a capability in terms of a system-of-systems construct. To further define and validate CE, a formal evaluation strategy has been undertaken to assess the three main axes of CapDEM: People, via a Capability Engineering Team; Process, via a Capability Engineering Process; and Materiel, via a Collaborative Engineering Environment. Based on an iterative approach, the evaluation will spiral through three developmental iterations: Exercises Alpha, Beta and Gamma. These exercises will help to identify the refinements needed to enable a rigorous CE process, along with an engineering team who will be engaged in the process, and a technological infrastructure that will facilitate its development and execution.

Résumé

Le ministère de la Défense nationale et les Forces canadiennes mettent actuellement en place une approche de planification axée sur les capacités (PAC) comme élément central de leur processus d'affaire global. Le projet de démonstration technologique « Définition, ingénierie et gestion des capacités collaboratives » (DIGCap) vise à définir et valider l’ingénierie des capacités (IC) comme nouveau moyen de soutenir la PAC. On vise à introduire la rigueur de l’ingénierie système en la transposant dans le paradigme de système de systèmes. Pour faire progresser et valider l’IC, une stratégie formelle d’exercices est nécessaire afin d’évaluer les trois axes de DIGCap : l’axe du personnel via une équipe d’ingénierie des capacités; l’axe du processus via un processus d’ingénierie des capacités et l’axe du matériel via un environnement collaboratif d’ingénierie. Basé sur une approche itérative, l'évaluation se divise en trois itérations : les exercices alpha, bêta et gamma. Ces exercices contribueront à identifier les améliorations nécessaires pour obtenir un processus rigoureux impliquant une équipe d’ingénierie engagée et une infrastructure technologique appropriée.

DRDC Valcartier TM 2005-164 i

This page intentionally left blank.

ii DRDC Valcartier TM 2005-164

Executive summary

The Department of National Defence (DND) and the Canadian Forces (CF) are currently implementing Capability Based Planning (CBP) as a core element of their overall business process. According to CBP, once a capability is defined, it must be both properly engineered and managed. To this end, the Collaborative Capability Definition, Engineering, and Management (CapDEM) Technology Demonstration project aims to define Capability Engineering (CE) and validate the discipline in the Canadian defence context, including collaboration with various DND and industry stakeholders. This memorandum introduces the strategy undertaken to evaluate the “CapDEM approach” - a necessary step prior to any commitment to institutionalize CE within the DND/CF.

The strategy is organized along CapDEM’s three axes: People, Process and Materiel.

• People: The Capability Engineering Team (CET) concept represents a cross-functional, multidisciplinary team committed to applying and managing the CE process to help support Capability Based Planning.

• Process: The Capability Engineering Process (CEP) is an iterative and incremental process starting from capability shortcomings and finishing with a set of Investment Options and a recommendation in order to provide senior DND/CF personnel with relevant information for enhanced decision-making capabilities.

• Materiel: The Collaborative Engineering Environment (CEE) consists of a set of tools, technologies and facilities that will help facilitate information exchange and collaboration among engineers, subject matter specialists and managers at multiple, geographically-distributed locations for the purpose of defining, developing and evaluating a capability.

Capability engineering is a proposed concept that is intended to support CBP by utilizing engineering rigour in the realization of a capability within a system-of-systems construct. The approach and methodology for the evaluation strategy utilizes similar rigour in its use of an iterative development approach, in which each iteration incorporates an increasingly encompassing and detailed subset of CEP activities. Each iteration employs detailed planning, experimentation and analysis of results. Using a spiral methodology, the evaluation will cycle through three iterations: Exercise Alpha, Exercise Beta and Exercise Gamma. The exercises will be carried out over an 18 month period ending in mid-2006. The results of each iteration will be used to improve and/or clarify the elements of each CapDEM axis and collect lessons learned about each axis, including any difficulties that may be encountered by the participants. This permits the gradual experimentation, assessment and evaluation of each aspect of the team’s roles and responsibilities, the process activities and deliverables, and the collaborative tools and facilities required for capability engineering.

The scope of the three exercises may be described as follows:

DRDC Valcartier TM 2005-164 iii

• Exercise Alpha will be the initial effort to evaluate the interaction between the three CapDEM axes. Based on standard experimental approaches and using a “controlled” environment, this exercise is designed to be a “proof-of-concept” trial in how to integrate and evaluate the interdependencies of the three main CapDEM axes. The primary goal is to “debug” the evaluation methodology before proceeding with larger experimental subject groups in subsequent exercises.

• Exercise Beta is intended to be a complete functional test and evaluation of CapDEM’s CEP, CET and CEE axes. Based on a realistic problem definition, Exercise Beta shifts emphasis from debugging the evaluation methodology towards the core of the problem and CapDEM’s ability to meet the client’s needs.

• Exercise Gamma is intended to be a complete “3rd party” functional test and evaluation of CapDEM’s CEP, CET and CEE axes. Based on a realistic problem definition, Exercise Gamma shifts emphasis towards external groups being able to address their problem using the CapDEM approach.

This evaluation strategy will involve the various stakeholders and future owners of the process to help enable a robust capability engineering construct as a key component of the Capability Management Framework.

Robbins, W., Waruszynski, B., Lalancette, C., Lizotte, M., Nécaille, C. 2006. CapDEM evaluation strategy: Integrating people, process and materiel. DRDC Valcartier TM 2005-164. Defence Research and Development Canada.

iv DRDC Valcartier TM 2005-164

Sommaire

Le ministère de la Défense nationale (MDN) et les Forces canadiennes (FC) mettent actuellement en place une approche de planification axée sur les capacités (PAC) comme élément central de leur processus d'affaire global. Les principes de la PAC indiquent, qu’une fois définie, une capacité doit être mise en place et gérée adéquatement. Le projet de démonstration technologique « Définition, ingénierie et gestion des capacités collaboratives » (DIGCap) vise à définir et valider l’ingénierie des capacités (IC) dans le cadre de la défense canadienne, incluant la collaboration des différents partenaires du MDN et de l’industrie. Ce mémorandum présente la stratégie proposée pour valider « l’approche DIGCap » - une étape essentielle avant de s’engager fermement à institutionnaliser l’IC.

Cette stratégie est organisée selon les trois axes de DIGCap : le personnel, le processus et le matériel.

• Le personnel : L’équipe d’ingénierie des capacités (EIC) est multidisciplinaire. Elle possède des compétences transversales et complémentaires. Son rôle est d’appliquer et de gérer le processus d’ingénierie des capacités qui soutient la PAC.

• Le processus : Le processus d’ingénierie des capacités (PIC) est un processus incrémental et itératif. À partir d’une ou de plusieurs lacunes constatées pour une capacité, le PIC produit un ensemble d’options d’investissement et une recommandation destinées à fournir aux gestionnaires supérieurs du MDN et des FC de l’information pertinente améliorant le processus décisionnel relatif aux capacités.

• Le matériel: L’environnement collaboratif d’ingénierie (ECI) est un ensemble d’outils, de technologies et d’installations destiné à faciliter l’échange d’informations et à favoriser la collaboration entre les différents acteurs (analystes, ingénieurs, architectes, spécialistes et gestionnaires) de l’ingénierie des capacités qui oeuvrent à partir d’endroits géographiquement distribués.

Le concept d’ingénierie des capacités est proposé dans le but de soutenir la PAC. Il utilise la rigueur de l’ingénierie et la transpose dans le paradigme des systèmes de systèmes. La stratégie d’évaluation choisie se base sur une approche itérative. Chaque itération comprend un sous-ensemble des activités du PIC. Elle fait l’objet d’une planification détaillée et comprend une phase d’expérimentation et d’analyse des résultats. L’approche est composée de trois itérations: les exercices alpha, bêta et gamma. Ces exercices se dérouleront sur une période de 18 mois se terminant à l’été 2006. Les résultats de chaque itération seront exploités pour améliorer chacun des axes du DIGCap et en tirer des leçons telles que les difficultés rencontrées par les participants. Chacun des aspects sera ainsi graduellement évalué et raffiné, qu’il s’agisse des rôles et responsabilités de l’EIC, des différentes activités et livrables décrits dans le PIC ou de la convivialité et des fonctionnalités offertes par l’ECI.

DRDC Valcartier TM 2005-164 v

La portée des exercices se décline comme suit:

• L’exercice Alpha constitue le premier effort pour évaluer les interactions entre les trois axes de DIGCap et mettre au point l’environnement d’évaluation. Il constitue une preuve de concept basé sur une approche expérimentale classique, pour évaluer et raffiner l’intégration des trois axes de DIGCap.

• L’exercice Bêta vise une évaluation et un test fonctionnel complet des trois axes de DIGCap. Par rapport à Alpha, l’emphase de Bêta se déplace de la méthode d’expérimentation vers le but réel de DIGCap, soit de satisfaire les besoins du MDN et des FC.

• L’exercice Gamma vise également une évaluation et un test fonctionnel complet des trois axes mais par une équipe indépendante de DIGCap. Par rapport à Bêta, l’emphase de Gamma se déplace des besoins du MDN et des FC à la capacité des parties prenantes de résoudre elles-mêmes leur problème en utilisant l’approche DIGCap. Elle cible l’acceptation de l’approche par les parties prenantes.

Cette stratégie d’évaluation impliquera différentes parties prenantes dont les futurs propriétaires du processus. Elle contribuera ainsi à mettre en place une ingénierie des capacités robuste jouant un rôle important dans la PAC.

Robbins, W., Waruszynski, B., Lalancette, C., Lizotte, M., Nécaille, C. 2006. CapDEM evaluation strategy: Integrating people, process and materiel. RDDC Valcartier TM 2005-164. R & D pour la défense Canada.

vi DRDC Valcartier TM 2005-164

Table of contents

Abstract/Résumé.......................................................................................................................... i

Executive summary ................................................................................................................... iii

Sommaire.................................................................................................................................... v

Table of contents ...................................................................................................................... vii

List of figures .......................................................................................................................... viii

List of tables ............................................................................................................................ viii

1. Introduction ................................................................................................................... 1

2. The Three CapDEM Axes - People, Process and Materiel ........................................... 3 2.1 The People Axis: Capability Engineering Team ............................................. 4 2.2 The Process Axis: Capability Engineering Process......................................... 5 2.3 The Materiel Axis: Collaborative Engineering Environment.......................... 7

3. Evaluation Strategy ..................................................................................................... 10 3.1 Approach and methodology ........................................................................... 10 3.2 Roles and responsibilities ............................................................................... 12

4. Exercise Description.................................................................................................... 14 4.1 Exercise Alpha: “Debugging” the methodology ........................................... 14 4.2 Exercise Beta: Solving a “real problem”....................................................... 18 4.3 Exercise Gamma: Involving a bona fide client ............................................. 23

5. Conclusion................................................................................................................... 29

6. References ................................................................................................................... 30

List of acronyms ....................................................................................................................... 31

Distribution list ......................................................................................................................... 33

DRDC Valcartier TM 2005-164 vii

List of figures

Figure 1. The Three Axes of CapDEM ..................................................................................... 3

Figure 2. The Capability Engineering Process .......................................................................... 5

Figure 3. CEP process efforts by stage...................................................................................... 6

Figure 4. Stages of the Capability Engineering Process............................................................ 7

List of tables

Table 1. Collaborative Engineering Environment ..................................................................... 8

Table 2. Experimentation work plan and tentative timeline.................................................... 11

Table 3. Scope of Exercise Alpha ........................................................................................... 17

Table 4. Scope of Exercise Beta .............................................................................................. 21

Table 5. Scope of Exercise Gamma......................................................................................... 27

viii DRDC Valcartier TM 2005-164

1. Introduction

The Department of National Defence (DND) and the Canadian Forces (CF) are currently implementing Capability Based Planning (CBP) [1] as a core element in their overall business process. According to the basics of CBP, a capability, once defined, must be both properly engineered and managed. To that end, the Collaborative Capability Definition, Engineering, and Management (CapDEM) Technology Demonstration (TD) project aims to define and validate Capability Engineering (CE) [2] within the Canadian defence context, including collaboration with various DND and industry stakeholders. Based on the scope and potential implications of employing CE within DND, it is essential to properly evaluate its approach and minimize any associated risks prior to any commitment to institutionalize this new construct. Consequently, a formal evaluation strategy for CapDEM is as fundamental as a testing strategy within classical systems engineering.

This report introduces the strategy undertaken to evaluate the three fundamental axes of CapDEM (i.e., People, Process and Materiel) as realized via the Capability Engineering Process (CEP) [3] and its two essential enablers: the Capability Engineering Team (CET) and the Collaborative Engineering Environment (CEE).

Based on an iterative approach, the evaluation spirals through three developmental iterations (Exercises Alpha, Beta and Gamma) and will examine the three main CapDEM axes (i.e., People, Process and Materiel) via increasing scope and resolution. Detailed planning, experimentation and analysis of the results will occur in each exercise. The first iteration (Exercise Alpha) aims mainly to “debug” the methodology (i.e., solve any substantive problems) with people closely involved in CapDEM. The second iteration (Exercise Beta) introduces appropriate stakeholders as participants and uses a realistic problem scenario. The third iteration (Exercise Gamma) aims to prepare for institutionalization by providing external validation of the CapDEM approach through the use of a larger, more involved problem definition and the sole use of non-CapDEM, external experimental subjects.

Although the CEP is a primary focus of CapDEM, it is recognized that process alone does not generate a successful program. In particular, an engineering team, who will be engaged in the process, and a technological infrastructure, which will facilitate its development and execution, both play fundamental roles in the successful realization of CapDEM. Therefore, the basic components of the CapDEM approach can be described along three axes - People, Process and Materiel:

• People: The CET concept represents a cross-functional, multidisciplinary team (with complementary skills) committed to applying and managing the capability engineering process to help support capability-based planners. The CET concept incorporates the best practices and lessons learned in teamwork, which in turn will help to further augment the vision and development of a future team concept - Integrated Capability Engineering Teams (ICE-Ts).

DRDC Valcartier TM 2005-164 1

• Process: The CEP is an iterative and incremental process starting from capability shortcomings and finishing with a set of Investment Options among which one is recommended. These options will provide senior DND/CF personnel with relevant information for enhanced decision-making capabilities.

• Materiel: The CEE consists of a set of tools, technologies and facilities that will help enable information exchange and collaboration among engineers, subject matter specialists and managers at multiple, geographically-distributed locations for the purpose of defining, developing and evaluating a capability.

Towards this end, this technical memorandum outlines a strategy for assessing the three main axes of CapDEM. It describes the approach and methodology for conducting the process evaluation; the roles, responsibilities and teamwork principles of the CET; the materiel components that comprise the CEE; and the primary tasks involved in the three iterative cycles of the evaluation.

2 DRDC Valcartier TM 2005-164

2. The Three CapDEM Axes - People, Process and Materiel

As a means by which to deliver on the potential of Capability Based Planning, CapDEM is organized along three primary axes: People, Process and Materiel. In short, the process defines the rules and methodologies by which people apply their expertise, creativity and engineering knowledge, using the appropriate technology and tools to facilitate the development of the necessary capabilities (Figure 1).

This section provides a brief overview of the three axes in more detail, setting the stage for how they work together and their role in the evaluation strategy.

Figure 1. The Three Axes of CapDEM


2.1 The People Axis: Capability Engineering Team

The Capability Engineering Team construct represents a central and fundamental element of organizational design for the CapDEM TD project. The CET construct is defined as a cross-functional, multidisciplinary team (with complementary skills) committed to applying and managing the capability engineering process to help support CBP. The CET is basically composed of a team leader, systems engineers, systems architects and requirements/operational analysts. The vision is to evolve the CET construct to an Integrated Capability Engineering Team (ICE-T) construct [4], one that will be composed of a core analytical team, encompassing a combination of a team leader, systems engineers, systems architects, a lifecycle costing specialist, a human systems integration engineer and other operational analysts. This core analytical team will be partnered with operational subject matter experts (SMEs) and liaison members from across the PRICIE1 capability components.

The members of the CET are comprised of team leaders and individuals who bring their expertise, skills and knowledge to meet specific objectives and targets. Team leaders need to have the necessary skills for team management and leadership (e.g., teamwork, team development, team design and process, team motivation, team-building, mentoring, coaching, trust-building, communication and capability-based team management). The recruitment and selection of team members is based on team leaders who are empowered with greater authority, responsibility and accountability over the selection of team members, rating authority and rotation of members. As a result, the level of empowerment needs to be aligned with appropriate team leadership competencies and policies. Team skill requirements include: technical or functional expertise, problem-solving and decision-making skills and interpersonal skills (with a focus on personal learning and development).

To ensure a coordinated and multidisciplinary approach to the formation of the Capability Engineering Team, a CET-specific Charter is proposed to make certain that all of the elements have been considered for achieving high-performance teams. The CET Charter is designed to promote organizational support and commitment for effective capability engineering. This Charter outlines a core CET that is responsible for the development and implementation of the capability engineering construct within CPB. Streamlining coordination and communication within the CET construct helps to promote a better understanding of the objectives, issues and milestones. Enhanced coordination and communication also creates a stronger sense of empowerment, ownership, responsibility and accountability among team members, and ensures that the team will run successfully and synergistically within and across all work streams.

1 PRICIE = Personnel, Research, Infrastructure & Organization; Concepts, Doctrine & Collective Training; IT Infrastructure; Equipment, Supplies and Services.


2.2 The Process Axis: Capability Engineering Process

This section provides an overview of the Capability Engineering Process. A more detailed discussion can be found in a number of CapDEM documents, including [3], [5], [6] and [7].

Illustrated in Figure 2, the CEP is positioned below the Enterprise architectural level and organized into two process groups: a set of Management processes and a set of Technical processes. The two sets of processes operate in parallel. They produce their deliverables iteratively and incrementally throughout the whole CEP. The application of the CEP in a particular situation is known as an “instance” of the CEP. Each instance of the CEP will be initiated with the identification of a capability’s shortcomings and will end with the delivery of a set of Investment Options and a recommendation to address the shortcomings. The Investment Options provide DND/CF decision-makers with information to be considered when making decisions, with specific attention to issues of cost, risk and/or feasibility in relationship to one or more facets of the PRICIE construct.

Figure 2. The Capability Engineering Process


The CEP Management process set ensures the proper planning, assessment and control of the Technical process set. The Management process set is also responsible for maintaining a shareable, configuration-managed repository of artifacts that can be used for tracking, re-use and reporting purposes. The CEP Management processes interact with DND and government processes for all matters concerning mandate, requirements in terms of resources, services and infrastructure, along with all decisions that cannot be made at CEP level.

The Technical process set applies the rigour of systems engineering to analyze a capability’s shortcomings, define possible Investment Options, update Operational and System-of-Systems (SoS) architectures, and ultimately package those Investment Options. The heart of the Technical process set consists of two loops (the Requirements Analysis loop and the Design loop) that are used to produce a set of Investment Options bridging the gap between “as-is” and “to-be” architectures.

In addition to processes, CEP uses the notion of stages. It uses the following stages:

• Inception

• Analysis

• Architecting

• Options Definition

It is important to differentiate CEP “stages” from CEP “processes”. The various stages structure and organize the lifecycle of a CEP instance, each with specific deliverables/approval gates; on the other hand, CEP processes result in the realization of a deliverable, often performed iteratively and independently within each stage. Figure 3 shows the proportion of process efforts for each stage.

Figure 3. CEP process efforts by stage

InceptionInception AnalysisAnalysis ArchitectingArchitecting Options Def.Options Def.

Main deliverables:Requirements

Operational Architecture

SoS Architecture

Investment Options

Main deliverables:Requirements

Operational Architecture

SoS Architecture

Investment Options

Processes:Project Management

Analyse Capability Gap

Update Operational Architecture

Update SoS Architecture

Define Investment Options

Processes:Project Management




Define Investment Options


Figure 4 shows the sequence of stages from the initial state (solid dot) to the final state (solid dot surrounded by a circle) of a CEP instance.

Figure 4. Stages of the Capability Engineering Process

These stages are defined along the CEP deliverables: Project Plan, Requirements Baseline, Operational Architecture, SoS Architecture, and Options Definition. The transition between the various stages takes place at the first approval of a specific deliverable. These are as follows:

• Inception to Analysis: first approval of the Project plan

• Analysis to Architecting: first approval of a Requirements Baseline

• Architecting to Options Definition: first approval of both architectures (Operational and SoS)

Within the Architecting stage, there are at least 3 sub-stages called increments. The transition from one increment to another happens when feedback/comments (as opposed to approval) are received from key stakeholders. Substages can also occur within other stages as required.

2.3 The Materiel Axis: Collaborative Engineering Environment

Within CapDEM, the role of materiel is as an infrastructural enabler to people and process. Within this evaluation strategy, the focus on materiel is in terms of its realization as tools, technologies and facilities that provide functionality and services to CETs as they apply the CEP within the various exercises.

Organizationally, CapDEM’s materiel considerations are encompassed within a work stream known as the Collaborative Engineering Environment, or CEE [8]. The CEE is a logical environment consisting of a set of tools that facilitate information exchange and collaboration among engineers, subject matter specialists and managers at multiple, geographically-distributed locations for the purpose of defining, developing


and evaluating a capability. Primarily based on commercially available tools and applications (COTS), the function of such an environment is to enable project stakeholders to have a common location and a common interface by which to access information, utilize specialized applications and communicate with each other. Ultimately, the CEE is intended to promote holistic system thinking, i.e., enable stakeholders to understand how they relate to each other in the project, hence improving their relationship with the process and its application through more effective collaboration.

The CEE is organized from two orthogonal perspectives: (1) distributed deployment of the CEE toolset on different servers at different physical locations throughout the DRENet; and (2) in logical groupings based on tool focus to facilitate particular work domains and practices as well as administration, management, training and various support issues. The first perspective is important in terms of dealing with actual day-to-day management and access/security of tools with respect to deployment and the various users on the different networks employed by CapDEM participants. The second perspective organizes the tools into focus areas: (1) the Integrated Engineering Environment (IEE), which consists of domain-specific tools used for a specialized function or purpose; (2) the Integrated Collaborative Environment (ICE), which is the grouping of general-purpose and/or collaborative tools and facilities (i.e., those designed to facilitate work and working together); and (3) the Integrated Synthetic Environment (ISE), which is intended to address the use of modelling and simulation (M&S) technologies to validate, test and analyze proposed solutions. An example of an IEE tool would be a costing or network analysis tool while an ICE tool could range from a video teleconference (VTC) system to a document/information repository to instant messaging or email. ISE tools could include various virtual or constructive simulators as well as tools intended to facilitate simulation development.

Together, IEE, ICE and ISE form the larger CEE which is used to both do the work and facilitate participants’ work within any particular effort. Table 1 highlights the major elements of the current CEE; details of the various collaborative tools available via the ACCESS2 Labs and the Livelink Portal are omitted for brevity. For purposes of this effort, focus is primarily on the IEE and ICE components of the CEE. Evaluation of the ISE component will be addressed separately.

Table 1. Collaborative Engineering Environment

Tool Purpose Focus/Audience

DOORS Requirements analysis Managerial stakeholders

CORE Functional analysis Systems engineering/architecture SMEs

OpNet Network analysis Communications/network SMEs

IPME Human systems integration Human factors SMEs

2 ACCESS is the new (proposed) naming standard for the collaborative lab facilities previously referred to as ACCES (in Ottawa) and VOLCAN (in Valcartier) within other CapDEM documentation.


InterchangeSE Central (model) repository, data integration and management

Integrators, managers, systems engineers/architects

Livelink (Document) repository & collaborative portal

Enhanced file sharing, information exchange, hub for participant interaction

ACCESS Labs Audio conferencing / VTC and collaborative tools Collaborative labs/workspace

DRENet Email & tool host network Host network for IEE & ICE tools


3. Evaluation Strategy

The following section presents the evaluation strategy for the CapDEM axes, including discussion of the approach and methodology along with the roles and responsibilities of those involved.

3.1 Approach and methodology

The approach and methodology for the evaluation strategy is based on an iterative development approach, where each iteration will incorporate a subset of CEP activities. Each iteration will include detailed planning, the experimentation itself and analysis of results. This spiral approach will cycle through three developmental iterations, Exercise Alpha, Exercise Beta and Exercise Gamma, and will evaluate each of the three main axes: People, Process and Materiel. The core portion of the evaluation will be based on the process axis, which in turn, will be enabled by the people and materiel axes. The evaluation is planned to incorporate the “ISR for Maritime Environment” planning scenario for purposes of providing context within at least the first two exercises. A different one to be identified with the stakeholders may be used for the third exercise.

The proposed approach follows the tenets of good system development, including incremental growth and management of expectations. As such, the exercises represent a transposition of the testing activities in software-intensive systems: unit testing, functional testing and acceptance testing. The scope (number) and resolution (depth) of the elements to be evaluated will increase from one iteration to the next. Specifically, the second and third iterations will evaluate more elements than the first. Participant feedback will enable incremental improvements to the evaluation environment through enhancement of the measurement tools and the evaluation process after the first and second iterations. Time and effort will be allowed for in order to incorporate any necessary modifications before starting subsequent iterations, based on corrections to existing elements and creation of any additional (new) elements if needed. In summary, the results of an iteration will be used to improve the axes elements, collect lessons learned about the axes (i.e., elements that may be good, weak, missing and unclear), including any difficulties that may be encountered by the participants.

In addition, the CapDEM Guidebook will be used as a key integrated reference tool that will assist all key players in understanding the three CapDEM axes. The Guidebook will encompass the development of a “transformation toolkit” that to facilitate continuous improvement of the CET, CEP and CEE. For instance, in the CapDEM Guidebook, a Team Charter template will enable the team to learn and understand the intent, role and function of the CET, including how each member can benefit from participating in the work team construct (e.g., embedded team-building activities to ensure that team leaders and members continue to build and sustain


healthy team dynamics). This Guidebook will continue to evolve throughout the three exercises, combining the experiences and wisdom of the evaluation and subject teams.

Experimentation work plan

The experimentation will be carried out over 18 months and will end in mid-2006. A tentative work plan is proposed in Table 2. Each activity will be based on a specified period of time for conducting each exercise. The evaluation will incorporate a scenario, along with a description of the problem and a descriptive narration providing the background details to the overall exercise.

Data gathering and analysis

An appropriate selection of metrics will be used to measure the level of compliance of the three axes. The starting point for metric development will be based on the principles established in [5].

For each exercise, data will be collected through the use of standardized surveys and structured interviews. Additional interviews and observations will be incorporated into the analyses to help supplement the preliminary data findings. The data gathering techniques will be developed for each separate exercise and will be incorporated into the iterative development approach for evaluating the three axes.

Table 2. Experimentation work plan and tentative timeline

Activity Duration Proposed Time Period

Establish evaluation strategy 2 months January - March 2005

Exercise Alpha 2 months March - May 2005

Prepare environment 1.5 weeks

Coordinate participation 1 week

Run exercise 4 weeks

Evaluate results 1.5 weeks

Exercise Beta ~6 months May - November 2005

Prepare environment 3 weeks

Coordinate participation 2 weeks

Run exercise 4 months

Evaluate results 1 month

Exercise Gamma ~10 months November 2005 - September 2006

Prepare environment 2 months


Coordinate participation 1 month

Run exercise 6 months

Evaluate results 2 months

Produce final report ~2 months September - November 2006

The data analysis will be outlined for analyzing the results of the evaluation for each iterative cycle (exercise). Descriptive statistics will be provided for the survey portion of the evaluation. The content will be analyzed with a focus on common themes and issues.

3.2 Roles and responsibilities

In employing standard experiment design practices, the evaluation will employ two groups of people for each exercise: (1) a subject team; and (2) an experimentation team. The subject team corresponds to the CET that will execute each individual exercise. The roles and responsibilities of each team member of the CET will be outlined for each exercise based on the iterative development approach. The experimentation team will consist of observers who will perform the observation and evaluation of the exercise.

The Subject Team

The subject team will be selected on a per-exercise basis and be comprised of actors who will carry out specific roles within the CEP. A text description will be put forward to describe how the team was selected for each exercise. At that point, they will be referred to as the Capability Engineering Team. Specifically, the CET will:

• Have the skill sets needed to apply the process (across the three iterative exercises).

• Be developed outlining specific functional roles and responsibilities for each exercise.

• Be given a capability gap to solve, and will be asked to use the CEP to solve this gap throughout the three exercises.

• Employ the CEE to enable the CEP.

• Devise a Team Charter, which in turn, will incorporate CET principles and team-building guidelines. The CET principles will incorporate personnel issues which may ultimately impact on team performance (e.g., recruitment/selection, motivation, recognition and rewards, training and development, communication, performance, trust and workload). The Charter will be initially developed for the first exercise and will be developed further for the remaining two exercises.


• Provide feedback to the experimentation team regarding each exercise.

Construction of the Subject Teams

In formulating the various subject teams, the following guidelines will be observed:

• To ensure an optimal level of collaboration, the CET members will be comprised of an equivalent number of people from the two DRDC centres, at least for the first iteration exercise.

• Collaboration and teamwork represent two important principles in the CEP, including collaborative engineering and information sharing. Within the first exercise, the collaboration and teamwork principles will be incorporated into the CET Team Charter. Participants will also attend a short motivational presentation to outline the principles of teamwork and their practical realization as part of CapDEM and the evaluation exercises.

• The CET membership will be enhanced for each iterative exercise to maintain a level of continuity among team participants. The iterative development approach will ensure that team continuity is maintained between experiments.

• Team members will have the appropriate technical or functional expertise, problem-solving and decision-making skills, and interpersonal skills (with a focus on personal learning and development).

Experimentation team

The experimentation (evaluation) team will be comprised of various CapDEM representatives from each axis (People, Process and Materiel) who will serve as observers to assess the CET, CEP and CEE. Specifically, each representative of the experimentation team will be responsible for:

• conducting their portion of the assessment within each exercise;

• preparing the evaluation methodology and tools for each exercise (each representative will provide the tools and mechanisms to gather the data); and

• providing the instructions and overview of the CEP, CEE and CET (specifically, CET principles and team-building guidelines) to the Subject Team.

Client and stakeholder involvement

Clients and stakeholders will be encouraged to participate in the second and third exercises; however, the CEP evaluation will need to remain clear of any critical path of existing projects to maintain control over the exercise conditions, particularly in the second exercise.


4. Exercise Description

This section overviews the specifics of each of the three exercises that comprise the iterative evaluation strategy outlined in this paper. During the exercises, the activities to be conducted will be organized according to the standard CEP stages as described in section 2.2. They are used to specify when (i.e., at which stage) a specific axis item will be evaluated.

4.1 Exercise Alpha: “Debugging” the methodology

Exercise Alpha will be the initial effort to evaluate the interaction between the three CapDEM axes. Based on standard experimental approaches and using a “controlled” environment, this exercise is designed to be a “proof-of-concept” trial in how to integrate and evaluate the integration of the three main CapDEM tenets.

Goals

The primary goal of this iteration is to enable the evaluation team to “debug” the evaluation environment with a very limited set of experimental subjects. The intent is to ensure internal, self-consistency within the axes and the evaluation constructs before proceeding on to a larger experimental subject group (i.e., a broader group of CapDEM stakeholders). Doing so will also help ensure smoother execution of subsequent exercises along with increasing the credibility associated with their resultant observations.

In addition, specific objectives of this iteration for each CapDEM axis are as follows:

• People: Ensure the necessary team roles and responsibilities are identified along with the selection of available individuals to participate in the exercise itself. Ensure that the principles of teamwork and collaboration/information sharing are embedded into the CET (principles outlined and supported by them).

• Process: Ensure understanding of how to initiate each of the Management and Technical processes.

• Materiel: Ensure the basic functionality that is useful for the CET is available and functional at the required level.

Outputs and outcomes

In achieving the above goals, specific outputs have been identified for each axis. They are as follows:


• People

• Realize the experimental CET, including an understanding of roles and responsibilities.

• Use appropriate teamwork principles, collaborative tools and practices within the CET.

• Process

• Familiarize the Subject Team with the Management and Technical processes.

• Familiarize the Subject Team with the CEP deliverables.

• Identify documentation weaknesses (process and deliverable templates) to enable adequate application in the next iterations.

• Identify process elements that will require particular attention during the training, e.g., anticipated issues or items difficult to understand.

• Materiel

• Familiarize the Subject Team with the basic tools to be used for the evaluation exercises.

• Utilize an initial, pre-established workflow between the various tools.

• Demonstrate and validate the utility of the selected collaboration tools.

Participation

This exercise will be based on the internal CapDEM experimentation team acting out preliminary roles and responsibilities. A script will be created to outline a capability gap (realistic scenario). This script will be acted out by the CapDEM experimentation team.

• The primary roles and responsibilities will initially include:

• Project Manager/Chief Architect

• Requirements Engineer

• Business Architect

• Solutions Architect


• Collaboration and teamwork will be introduced at this point and will be reiterated throughout Exercises Beta and Gamma. A Team Charter and Teamwork Guidelines will be incorporated at the start (involving team-building exercises).

Training

A minimum of training will be given to exercise participants. Tentative training plans for this iteration are as follows:

• People: The Subject Team will participate in a short motivational presentation to outline the principles of teamwork and collaboration, and will be briefed on their participation as part of CapDEM and the evaluation exercises. This presentation will also involve the identification of roles and responsibilities of the CET membership.

• Process: In addition to the complete documentation of CEP V1, a short training session of four hours will be given to participants who are new to the process.

• Materiel: Various short introductory sessions will be conducted to ensure a basic level of comfort and familiarity with the selected tools for those participants not familiar with or trained on them.

Scope and effort

The intent of Exercise Alpha is to ensure that the evaluation of the CapDEM axes meet expectations and guidelines of both CapDEM and good scientific practice. As such, Exercise Alpha is intended to provide a “proving ground” to ensure that subsequent evaluation efforts will run smoothly. With this in mind, this exercise will be:

• Brief by design - that is, it is not to become a long, involved event;

• The actual duration is to be strictly (and substantially) less than the subsequent exercises.

The scope of each axis within Exercise Alpha is summarized in Table 3.


Table 3. Scope of Exercise Alpha

CEP Stage

CapDEM Axis

Inception Analysis Architecting Options Definition

Teamwork and collaboration • Form CET • Define team roles

& responsibilities • Introduce Team

Charter which will incorporate teamwork/ collaboration principles

• Evaluate teamwork and collaborative behaviours of team relative to Operational Architecture

• Update the Team Charter to incorporate the initial analysis

• Update selected portions of the Team Charter and teamwork/ collaboration principles

Team communication

People • Reduced team size • Internal CapDEM

working teams • Distributed work sites

• Establish communication mechanism(s)

• Evaluate efficacy of intra-team communication


• Update selected portions of the Team Charter and communication mechanisms

Plan, control and assert effort • Produce Project

plan (R1)3 • Update Project

plan (R2) • Update Project

plan (R3) Analyze Capability Gap

• Initiate study of capability gap

• Complete selected portions of Requirements document (R1)

• Update selected portions of Requirements document (R2)

• Update selected portions of Requirements document (R3)


• Initiate Operational

Architecture development

• Complete selected portions of Operational Architecture (R1)

• Update selected portions of Operational Architecture (R2)


• Initiate SoS


• Complete selected portions of SoS Architecture (R1)

• Update selected portions of SoS Architecture (R2)

Define Investissement Options

Process • Create key

deliverables with limited content using CEP V1 documents

• Realize at least a second release of each selected deliverable

• Initiate Options definition

• Refine Options definition

• Refine Options definition

• Complete selected portions of Options Vol 1 (R1)

• Complete selected portions of Options Vol 2 (R1)

ACCESS Labs • Audio/video

conferencing • Collaborative tools

and workspace

• Audio/video conferencing

• Collaborative tools and workspace





Livelink

Materiel • Use restricted tool set

(select functionality) • Focus on collaborative

aspects • Use of pre-established

IEE workflow

• Create project & folders

• Deposit CEP input document

• Share useful documents

• Check-in/check-out working document





3 “R” in parentheses indicates the “Release” number of the deliverable.


CEP Stage

CapDEM Axis


DOORS • Creation of

requirements • Update

Requirements • Update


Requirements CORE

• Create Operational Architecture

• Update Operational Architecture


Interchange • Create engineering

data repository

• Update engineering data repository



4.2 Exercise Beta: Solving a “real problem”

Exercise Beta is intended to be a complete functional test and evaluation of CapDEM’s CEP, CET and CEE axes. Based on a realistic problem definition, Exercise Beta shifts emphasis from debugging the evaluation methodology towards the “core” of the problem and CapDEM’s ability to meet its needs.

This exercise will involve a complete cycle of the CEP using the complete CEE. The CET will be composed of the previous (Exercise Alpha) team (to benefit from their experience) along with additional internal CapDEM participants to facilitate the enlarged scope of the exercise. Additionally, an appropriate client and select stakeholders will be involved in a limited manner, such that they will provide the entry point to the exercise (via a realistic capability gap) and will be given a chance to evaluate and criticize the process and its outputs.

Goals

The primary goal of this iteration is to test and evaluate the CEP via an instance based on a realistic problem. The capability gap to be addressed will be provided by the client, who will input a problem statement and receive a proposed solution as an output. As part of the effort, the pseudo-client will have the opportunity to evaluate the process and its outputs. The intent is to ensure that the necessary people, process and materiel can address a “real world” problem level, while still running the exercise in a semi-controlled environment (i.e., the subject group and timeline will still be controlled internally). This transitional exercise allows a gradual, controlled growth from one exercise to another, facilitating increased credibility in the evaluation team and its strategy along with the ability to more definitively examine the issue of scalability in application of the CapDEM axes.


• People: Test and evaluate the efficacy and appropriateness of the roles and responsibilities, and the principles of teamwork and collaboration.


• Process: Test and evaluate the complete set of CEP activities and deliverables (both Management and Technical). Evaluate the management process, which will be formally applied in this iteration.

• Materiel: Test and evaluate the currently deployed CEE (tools, technologies and facilities). Investigate which functionalities would seem essential for the CEP as well as which functionalities (tools, technologies or facilities) are missing and/or insufficient.



• People

• Review and identify any issues surrounding the specified roles and responsibilities of the CET.

• Review and evaluate the principles of teamwork and collaboration and the efficacy of team-building exercises.

• Finalize the Team Charter, including teamwork principles.

• Process

• Ensure the Subject Team clearly understands the Management and Technical processes.

• Ensure the Subject Team.clearly understands the CEP deliverables.

• Identify documentation weaknesses (process and deliverable templates) to facilitate application in subsequent iterations.

• Identify process elements that will require particular attention during training, for example, anticipated issues or items difficult to understand.

• Materiel

• Identify missing functionalities to more fully enable the CEP and collaboration inside the CET.

• Refine and validate the existing workflow between the various tools.

• Create a database of “Frequently Asked Questions” (FAQs), “Tips & Tricks” and lessons learned.


Participation

This iteration will be an evolution from the previous exercise (such that the experimentation team will utilize the best practices and lessons learned and apply them to this exercise). Specific participation details are as follows:

• The team will be comprised of eight to ten people, including clients and stakeholders.

• The previous roles will incorporate three additional roles. Therefore, the CET will be comprised of:



• Business Architect (Process and HR/Org Development)

• Solutions Architect (Applications and Systems)

• Technology Architect (Network and Security)

• Data Architect (Data and Information)

• M&S Architect (M&S and VV&A)

Training

A minimum of training will be given to exercise participants. This will be more comprehensive for new participants while former participants will be updated according to the expected goals of the exercise and the enlarged scope of each axis.

Tentative training plans for this iteration are as follows:

• People: The Subject Team will participate in a short motivational presentation to outline the principles of teamwork and collaboration, as well as their practical realization as part of the CapDEM and the evaluation exercises. Also included will be a project kick-off meeting along with select team-building exercises to promote team dynamics and communication.

• Process: A short training session of four hours will be given to new participants, while a shorter crash course (1 to 2 hours) will be offered to returning participants. This training will reflect the current version of the CEP at the time the exercise is conducted (due to its evolution since the Alpha exercise). Complete documentation on the current CEP will also be provided.

• Materiel: Various short introductory sessions will be conducted to ensure a basic level of comfort and familiarity with the selected tools for those participants not familiar with or trained on them. Specific attention will be given to:


• Providing a guideline on the essential functionalities;

• Establishing standard procedures for using the collaborative tools (along with alternative solutions).

Scope and effort

The intent of Exercise Beta is to ensure that the evaluation of the CapDEM axes meet the needs of a realistic capability engineering problem. As such, Exercise Beta is intended to move beyond the “proving ground” of Exercise Alpha to utilize externally provided input and offer outputs for external critique. With that in mind, this exercise will be:

• Medium length by design - that is, it will be longer than the initial exercise but still limited and controlled as part of the experimental design;

• Interaction with clients/stakeholders is to ensure relevance and to obtain useful feedback/critique; scheduling and flow, however, are still controlled internally and not by external entities.

In terms of effort, the activities to be conducted will be organized according to the standard CEP stages. The scope of each axis within Exercise Beta is summarized in Table 4.

Table 4. Scope of Exercise Beta

CEP

Stage

CapDEM Axis


Teamwork and collaboration • Form CET • Define team roles



• Define work directives



• Update selected portions of the Team Charter and teamwork / collaboration principles

Team communication

People • Define the roles and

responsibilities for the CET in balance with Team Charter

• Establish effective communication mechanisms in conjunction with Team Charter






CEP Stage

CapDEM Axis


Plan, control and assert effort • Create Project plan

(R1) • Create

Configuration Management plan (R1)

• Create Risk Management plan (R1)

• Update Project plan (R2)

• Update Configuration Management plan (R2)

• Update Risk Management plan (R2)


• Update Configuration Management plan (R3)



• Initiate study of capability gap

• Create Requirement document (R1)

• Update Requirement document at each iteration (R2, R3, R4)

• Produce Requirement document final review (R5)


• Initiate Operational


• Update Operational Architecture at each iteration (R1, R2, R3)

• Produce Operational Architecture final review (R4)


• Initiate SoS


• Update SoS Architecture at each iteration (R1, R2)

• Produce SoS Architecture final review (R3)


Process • Complete each CEP

deliverables • Complete at least 3

iterations in the architecting stage


• Create Options Vol 1 (R1)


• Update Options Vol 1 at each iteration (R2, R3, R4)


• Produce Options Vol 1 final review (R5)




and workspace





• Audio/video cconferencing


Livelink • Create project &

folders • Deposit CEP input

document











Requirements CORE




Materiel • Fully exploit CEE

functionalities.

Interchange


CEP Stage

CapDEM Axis


• Create engineering data repository




OPNET

• Create comms model

• Update comms model

IPME

• Identify key HIS elements

• Outline HIS model to address identified issues

• Update HIS model

4.3 Exercise Gamma: Involving a bona fide client

Exercise Gamma is intended to be a complete “3rd party” functional test and evaluation of CapDEM’s CEP, CET and CEE axes. Based on a realistic problem definition, Exercise Gamma shifts emphasis towards external groups being able to address their problem using the CapDEM approach.

This exercise will involve a complete cycle of the CEP using the complete CEE. The CET will be composed of the previous internal participants (Exercises Alpha and Beta), forming the evaluation team along with additional external client and stakeholder participants filling the subject roles. The client and stakeholders will be involved as the subject team, first by providing the entry point to the exercise (via a realistic capability gap) and ultimately by evaluating/criticizing the process and its outputs. Internal coaching will be provided as part of the training to ensure a successful transition into the final exercise. Key to this iteration, however, is that the external participants will be the actual subject team during this exercise, thus providing an example of how external participants can independently utilize CapDEM in addressing a capability engineering problem.

Goals

The primary goal of this iteration is to test and evaluate the CEP, CEE and CET as applied to a realistic problem by an external client. The capability gap to be addressed will be provided as input (in the form of a problem statement) from the client; output to the client will be a proposed solution. As part of the effort, the client will have the opportunity to execute the process and subsequently evaluate both the process and its outputs. The intent is to ensure the necessary people, process and materiel can address a “real world” problem when being executed by an appropriate real-world client group. This final exercise benefits from the gradual, controlled growth of the previous exercises, facilitating increased credibility in the evaluation team and its strategy along with the validation afforded by integrating an actual client-based problem and process team as part of the actual exercise execution.



• People: Test and evaluate the final iteration of the CET. Put forward best practices and lessons learned for a comprehensive Team Charter, including the proper identification of roles and responsibilities, dynamic teamwork and collaboration practices, and effective team communication practices and mechanisms.

• Process: Test and evaluate the final version of the CEP, including the complete set of its activities and deliverables (both Management and Technical).

• Materiel: Test and evaluate the revised CEE (tools and facilities). Investigate and determine a reasonable list of functionalities (required, preferred and optional) for CEE institutionalization.



• People

• Complete the Team Charter, incorporating roles and responsibilities, as well as teamwork and collaboration principles for the CET.

• Put forward recommended best practices for initiating a successful CET.

• Link successful team dynamics with CEE (e.g., ACCESS labs and Livelink) for enhanced communication practices.

• Process

• Ensure the Subject Team clearly understands Management and Technical processes.

• Ensure the Subject Team clearly understands CEP deliverables.

• Identify documentation weaknesses (process and deliverable templates) to facilitate application during institutionalization within the DND/CF.

• Identify training weaknesses to be addressed when providing future training to DND/CF personnel.

• Identify metrics to measure and facilitate continuous improvment of the CEP.

• Identify critical factors for successful implementation of the CEP.

• Materiel


• Provide a user-validated list of mandatory/preferred functionalities to fully enable the CEP and facilitate collaboration inside the CET.

• Refine and validate the existing workflow between the various tools

• Provide an updated and expanded database of “Frequently Asked Questions” (FAQs), “Tips & Tricks” and lessons learned.

Participation

This iteration will be an evolution from the previous exercise (such that the experimentation team will utilize the best practices and lessons learned and apply them to this exercise). Specific participation details are as follows:

• Eight to twelve team members, including client and stakeholders, will be involved in the final exercise.

• The following will be included in the CET:



• Business Process Architect

• Organizational and Human Resources Architect

• Solutions Architect (Applications and Systems)

• Technology Architect (Network)

• Technology Architect (Security)

• Data Architect (Data and Information)

• M&S Architect (M&S and VV&A)

• There will be a need to include a Joint Capability Decision-Maker, the Capability Manager and Enterprise Architect.

Training

Comprehensive training will be given to all exercise participants. Training will be more detailed than previous efforts because most, if not all, subject team participants will be new to CapDEM. In addition to training for each axis, integrated training sessions will also be used to introduce a global perspective to members of the Gamma CET.


Based on training provided under Exercise Beta, tentative plans for this iteration’s training are as follows:

• People: In the final exercise, the Subject Team will participate in a short motivational presentation to outline their role and responsibilities, the principles of teamwork and collaboration, and their practical realization as part of the CapDEM and the evaluation exercises. Also included will be a project kick-off meeting along with selected team-building exercises to foster team dynamics, collaboration and team communication. The final Team Charter will incorporate the best practices in teamwork, collaboration and team communication.

• Process: A training session of at least four (4) hours will be given to ensure participants understand and can apply the CEP. This training will reflect the current version of the CEP at the time the exercise is conducted (due to its evolution since Exercise Beta).

• Materiel: Various sessions will be conducted to ensure a basic level of comfort and familiarity with the selected tools for those participants not familiar with or trained on them. Specific attention will be given to:

• Providing a guideline on the essential functionalities;

• Establishing standard procedures for using the collaborative tools (along with alternative solutions).

• Integrated Training: After an introduction to the basics of each axis, integrated training across all axes will be offered to show their synergies; length of time is likely to be approximately 3-4 days.

Scope and effort

The intent of Exercise Gamma is to ensure that the evaluation of the CapDEM axes meet needs of a realistic capability engineering problem as usable by an external, third-party client. As such, Exercise Gamma is intended to move beyond the more controlled environments of exercises Alpha and Beta to utilize externally provided input and output as well as utilizing external process executors. As such, the scope of the problem will still be limited to something tractable and practical in terms of the CapDEM project timeline.

Within this in mind, this exercise is to be:

• Longer-term by design - that is, it will be the longest of the three iterations;

• More involved in terms of client/stakeholder interaction - scheduling by the evaluation team will be at a macro level, such as the overall length of time allowed to execute the exercise; conversely, day-to-day scheduling will be done by the external client as part of the subject team.


In terms of effort, the activities to be conducted will be organized according to the standard CEP stages. The scope of each axis within Exercise Gamma is summarized in Table 5.

Table 5. Scope of Exercise Gamma

CEP Stage

CapDEM Axis


Teamwork and collaboration • Define team roles



• Define work directives


• Update the Team Charter to incorporate the analysis of the exercises

• Update selected portions of the Team Charter and teamwork / collaboration principles

Team communication

People • Define the roles and

responsibilities for the CET in balance with Team Charter

• Establish effective teamwork and collaboration principles for CET

• Establish effective communication mechanisms in conjunction with Team Charter





Plan, control and assert effort • Create Project plan

(R1) • Create Conf.

Management plan (R1)

• Creat Risk Management plan (R1)


• Update Conf. Management plan (R2)


• Update project plan (R3)

• Update Conf. Management plan (R3)


Analyse Capability Gap • Initiate study of

capability gap • Create

Requirement document (R1)

• Update Requirement document at each iteration (R2, R3, R4)

• Produce Requirement document final review (R5)


• Initiate Operational Architecture development

• Update Operational Architecture at each iteration (R1, R2, R3)

• Produce Operational Architecture final review (R4)


• Initiate SoS


• Update SoS Architecture at each iteration (R1, R2)

• Produce SoS Architecture final review (R3)

Process • Complete application

of a CEP instance along with each CEP deliverable



CEP Stage

CapDEM Axis











and workspace







Livelink • Create project &

folders • Deposit CEP input

document











Requirements CORE




Interchange

• Create engineering data repository




OPNET

• Create comms model

• Update comms model

IPME

• Identify key HSI elements

• Outline HSI model to address identified issues

• Update HSI model

Additional Tools Identified in Previous Exercises

Materiel • Fully exploit CEE

functionalities. • Augment functionality

based on previous exercises

• As identified in previous exercises




5. Conclusion

Capability engineering is a proposed concept that is intended to support Capability Based Planning by providing engineering rigour to the development of a capability within a system-of-systems construct. This evaluation strategy will involve the various stakeholders and future owners of the process to help enable a robust capability engineering construct as a key component of the Capability Management Framework.

The proposed evaluation strategy sketches a roadmap and provides an integrated and consistent approach for the validation and assessment of the three CapDEM axes: People, via the Capability Engineering Team; Process, via the Capability Engineering Process; and Materiel, via the Collaborative Engineering Environment. The strategy includes three iterations (i.e., Exercises Alpha, Beta and Gamma) and will enable the experimentation team to assess and evaluate the axes at different levels of granularity and refinement (unit, functional and acceptance). These exercises will help to identify the refinements needed to enable a rigorous CE process, along with an engineering team that will be engaged in the process, and a technological infrastructure which will facilitate its development and execution.

The most critical success factor for this evaluation strategy is strong client and stakeholder commitment. As the evaluation progresses from one iteration to the next, the willingness and intent of subject participants (in particular for Exercise Gamma) will play a significant role in the accurate measurement of the CapDEM axes and their interaction within a Capability Engineering context. Of particular necessity are participants with an open mind and a willingness to try new approaches despite involvement in other efforts and previous practices. Natural resistance to change and any potential negative reaction to new ways of doing things are also important elements to manage.

By design, this strategy does not include a detailed implementation plan for each exercise. Therefore, the immediate next steps will be to formulate the necessary plans for each exercise, starting with Exercise Alpha. Each exercise plan will consider the issues highlighted during the previous one, addressing the specifics of carrying out exercise execution, ranging from problem definition, personnel assignment and tool selection to the measurements to be taken and mechanisms involved in doing so. It will be these metrics and the resultant observations that will help refine the way ahead by offering recommendations and corrective actions to help evolve the Capability Engineering construct.


6. References

1. National Defence, Capability Based Planning for the Department of National Defence and the Canadian Forces, May 2002, http://www.vcds.forces.gc.ca/dgsp/00native/rep-pub/cbpManual_e.pdf.

2. Pagotto, J. and Walker, R., “Capability Engineering - Transforming Defence Acquisition inCanada”, in Proc. of SPIE Defense & Security Symposium, Orlando, FL, USA, April 2004.

3. Bagnulo, A., Drouin, C.,Harvey, G., Leclerc, J., Sweetnam, P., Wellwood, M. (2005). Capability Engineering Process Version 1 (CEP V1) - Description. Contract Report, CGI Inc.

4. Waruszynski, B.T. (2004). Work Team Strategies: Enhancing Team Development and Organizational Performance. A Foundation Report for Integrated Capability Engineering Teams. TM 2004-225, Defence R&D Canada - Ottawa.

5. CapDEM CEP Team (Bernier, F., Couture, M., Dussault, G., Lalancette, C., Lam, S., Lizotte, M., and Dr Mokhtari, M.), and CGI Team (Bagnulo, A., Baldwin, P., Drouin, C., Harvey, G., Leclerc, J., Sweetnam, P., and Wellwood, M.). (2004). The Capability Engineering Process (CEP) Foundations. TM 2004-425, Defence R&D Canada - Valcartier.

6. CapDEM CEP Team (Mokhtari, M., Lizotte, M., Lam, S., Lalancette, C., Dussault, G., Couture, M., and Bernier, F.), and CGI Team (Bagnulo, A., Drouin, C., Harvey, G., Leclerc, J., Sweetnam, P., and Wellwood, M.). (2004). Capability Engineering Process Version 1 (CEP V1) - Background and Overview. TM 2004-426. Defence R&D Canada - Valcartier.

7. Lizotte, M., Bernier, F., Dr Mokhtari, M., Couture, M., Dussault, G., Lalancette, C., Lam, S., and Lemieux, F. (2004). Toward a Capability Engineering Process, in Proceedings of the 24th Army Science Conference (ASC’04), Orlando, FL, USA. November 29 - December 2, 2004.

8. Robbins, W., Lam, S. and Lalancette, C. (2005). Towards a Collaborative Engineering Environment to Support Capability Engineering. In Proceedings of the 2005 INCOSE International Symposium - Systems Engineering: Bridging Industry, Government and Academia. Rochester, New York: International Council on Systems Engineering.


List of acronyms

ACCESS Advanced Collaborative Capability Engineering Support System

CapDEM Collaborative Capability Definition, Engineering and Management

CBP Capability Based Planning

CE Capability Engineering

CEE Collaborative Engineering Environment

CEP Capability Engineering Process

CET Capability Engineering Team

CF Canadian Forces

DIGCap Définition, ingéniere et gestion des capacités collaboratives

DND Department of National Defence

DRDC Defence Research and Development Canada

DRENet Defence Research Establishment Network

ECI Environnement collaboratif d’ingénierie

EIC Équipe d’ingénierie des capacités

FAQ Frequently asked questions

FC Forces canadiennes

HR Human resource(s)

HSI Human System Integration

IC Ingénierie des capacités

ICE Integrated Collaborative Environment

ICE-T Integrated Capability Engineering Team

IEE Integrated Engineering Environment

ISR Intelligence, Surveillance and Recognition


ISE Integrated Synthetic Environment

M&S Modelling and Simulation

MDN Ministère de la Défense nationale

PAC Planification axée sur les capacités

PIC Processus d’ingénierie des capacités

PRICIE Personnel, Research, Infrastructure & Organization; Concepts, Doctrine & Collective Training; IT Infrastructure; Equipment, Supplies and Services

SME Subject matter expert

SoS System of Systems

TD Technology Demonstration

VV&A Validation, verification and assessment


Distribution list

INTERNAL DISTRIBUTION

1 - Director General

3 - Document Library

1 - Head, System of Systems

1 - LCol B. Carrier

1 - P. Charland

1 - M. Couture

1 - G. Dussault

1 - C. Lalancette (author)

1 - M. Lizotte (author)

1 - C. Nécaille (author)


EXTERNAL DISTRIBUTION

1 - DRDKIM (PDF file)

1 - Director General Research and Development Programs

1 - Assoc DGRDP

1 - Director, Science and Technology (C4ISR)

1 - DST C4ISR 2

1 - Director General Strategic Planning

1 - Director Defensc Analysis

1 - DDA 2

1 - DDA 3

1 - DDA 3-3

1 - DDA 2-5

1 – DDA, Cdr M. Aucoin

1 - Director, Force Planning and Program Coordination 6

1 - DFPPC 7

1 - DFPPC 8-3

1 - Director, Defence Management 4

1 - DDM - Harmonization

1 - Director, Joint Force Capabilities

1 - Director, Maritime Requirements Sea

1 - Director, Materiel Acquisition and Support Program

1 - Director, Project Delivery Operational Information Systems

1 - DRDC Ottawa, Director General

1 - DRDC Ottawa, Deputy Director General


1 - DRDC Ottawa, Head/FFSE

10 - DRDC Ottawa, J. Pagotto

1 - DRDC Ottawa, S. Lam

1 - DRDC Ottawa, W. Robbins (author)

1 - DRDC Ottawa, B. Waruszynski (author)

1 - DRDC Ottawa Library

1 - DRDC Toronto, K. Hendy

1 - DRDC Toronto, W. Dyck

1 - DRDC ORD, K. Wheaton

1 - DRDC ORD, M. Dixon

1 - Canadian Forces Experimentation Centre (CFEC), Commandant

1 - CFEC, Cdr Lightbody

1 - CFEC, Maj D. Fullerton

1 - CFEC, Cdr Prudat

1 - CFEC Synthetic Environment Coordiation Office (SECO), R. Elliott

1 - CFEC SECO, P. Gauvin

1 - Director, Enterprise Architecture - A/Deputy Director

1 - DEA 5

1 - DEA 5-2


dcd03f ver.(06-1999)

SANS CLASSIFICATION COTE DE SÉCURITÉ DE LA FORMULE

(plus haut niveau du titre, du résumé ou des mots-clefs)

FICHE DE CONTRÔLE DU DOCUMENT

1. PROVENANCE (le nom et l’adresse) RDDC Valcartier

2. COTE DE SÉCURITÉ (y compris les notices d’avertissement, s’il y a lieu) Sans classification

3. TITRE (Indiquer la cote de sécurité au moyen de l’abréviation (S, C, R ou U) mise entre parenthèses, immédiatement après le titre.) CapDEM evaluation strategy: Integrating people, process and materiel

4. AUTEURS (Nom de famille, prénom et initiales. Indiquer les grades militaires, ex.: Bleau, Maj. Louis E.) CapDEM Evaluation Team - par ordre alphabétique (centre et auteurs) RDDC Ottawa: Wayne Robbins Barbara Waruszynski, RDDC Valcartier: Claire Lalancette, Michel Lizotte, Christophe Nécaille

5. DATE DE PUBLICATION DU DOCUMENT (mois et année) August 2006

6a. NOMBRE DE PAGES 35

6b. NOMBRE DE REFERENCES 8

7. DESCRIPTION DU DOCUMENT (La catégorie du document, par exemple rapport, note technique ou mémorandum. Indiquer les dates lorsque le rapport couvre une période définie.)

Mémorandum Technique

8. PARRAIN (le nom et l’adresse)

9a. NUMÉRO DU PROJET OU DE LA SUBVENTION (Spécifier si c’est un projet ou une subvention) 15as (Numéro de projet)

9b. NUMÉRO DE CONTRAT

10a. NUMÉRO DU DOCUMENT DE L’ORGANISME EXPÉDITEUR TM 2005-164

10b. AUTRES NUMÉROS DU DOCUMENT

N/A

11. ACCÈS AU DOCUMENT (Toutes les restrictions concernant une diffusion plus ample du document, autres que celles inhérentes à la cote de sécurité.)

Diffusion illimitée Diffusion limitée aux entrepreneurs des pays suivants (spécifier) Diffusion limitée aux entrepreneurs canadiens (avec une justification) Diffusion limitée aux organismes gouvernementaux (avec une justification) Diffusion limitée aux ministères de la Défense Autres

12. ANNONCE DU DOCUMENT (Toutes les restrictions à l’annonce bibliographique de ce document. Cela correspond, en principe, aux données d’accès au document (11). Lorsqu’une diffusion supplémentaire (à d’autres organismes que ceux précisés à la case 11) est possible, on pourra élargir le cercle de diffusion de l’annonce.)

SANS CLASSIFICATION

COTE DE LA SÉCURITÉ DE LA FORMULE (plus haut niveau du titre, du résumé ou des mots-clefs)

dcd03f ver.(06-1999)

SANS CLASSIFICATION COTE DE LA SÉCURITÉ DE LA FORMULE


13. SOMMAIRE (Un résumé clair et concis du document. Les renseignements peuvent aussi figurer ailleurs dans le document. Il est souhaitable que le sommaire des documents classifiés soit non classifié. Il faut inscrire au commencement de chaque paragraphe du sommaire la cote de sécurité applicable aux renseignements qui s’y trouvent, à moins que le document lui-même soit non classifié. Se servir des lettres suivantes: (S), (C), (R) ou (U). Il n’est pas nécessaire de fournir ici des sommaires dans les deux langues officielles à moins que le document soit bilingue.)

Le Ministère de la Défense nationale et les Forces canadiennes mettent actuellement en place une approche de planification axée sur les capacités (PAC) comme élément central de leur processus d'affaire global. Le projet de démonstration technologique « Définition, ingénierie et gestion des capacités collaboratives » (DIGCap) vise à définir et valider l’ingénierie des capacités (IC) comme un nouveau moyen pour supporter la PAC. On vise à introduire la rigueur de l’ingénierie système en la transposant dans le paradigme de système de systèmes. Pour faire progresser et valider l’IC, une stratégie formelle d’exercices est nécessaire afin d’évaluer les trois axes de DIGCap : l’axe du personnel via une équipe d’ingénierie des capacités; l’axe du processus via un processus d’ingénierie des capacités et l’axe du matériel via un environnement collaboratif d’ingénierie. Basé sur une approche itérative, l'évaluation se divise en trois itérations : les exercices alpha, bêta et gamma. Ces exercices contribueront à identifier les améliorations nécessaires pour obtenir un processus rigoureux impliquant une équipe d’ingénierie engagée et une infrastructure technologique appropriée.

14. MOTS-CLÉS, DESCRIPTEURS OU RENSEIGNEMENTS SPÉCIAUX (Expressions ou mots significatifs du point de vue technique, qui caractérisent un document et peuvent aider à le cataloguer. Il faut choisir des termes qui n’exigent pas de cote de sécurité. Des renseignements tels que le modèle de l’équipement, la marque de fabrique, le nom de code du projet militaire, la situation géographique, peuvent servir de mots-clés. Si possible, on doit choisir des mots-clés d’un thésaurus, par exemple le “Thesaurus of Engineering and Scientific Terms (TESTS)”. Nommer ce thésaurus. Si l’on ne peut pas trouver de termes non classifiés, il faut indiquer la classification de chaque terme comme on le fait avec le titre.)

Acquisition Process Capability Capability Engineering Capability Engineering Process Evaluation Strategy CapDEM Exercises CapDEM

SANS CLASSIFICATION

COTE DE SÉCURITÉ DE LA FORMULE


Canada’s Leader in Defenceand National Security

Science and Technology

Chef de file au Canada en matièrede science et de technologie pourla défense et la sécurité nationale

WWW.drdc-rddc.gc.ca

Defence R&D Canada R & D pour la défense Canada

capdem evaluation strategy - defence research …cradpdf.drdc-rddc.gc.ca/pdfs/unc83/p525740.pdf ·...

Documents