best practice supporting materials...best practice #1 * a red team must be able to operate...

14
Best Practice Supporting Materials Ratings for Candidate Best Practices (CBPs) Overall Rating: * Weakly Supported – Widely proposed as a best practice, but little to no empirical evidence, or empirical evidence that is contradictory. ** Moderately Supported – Some, but not conclusive empirical evidence in favor of the best practice, or the scope of the best practice is limited to a narrow context, such as one type of Red Teaming only. *** Strongly Supported – Multiple streams of empirical evidence from a number of Red Teaming contexts argue in favor of the best practice. Validation Rating: 0 = No validation; CBP is based on speculation alone. 1 = CBP has been derived based on the conduct / outcome of a single simulation/exercise. 2 = CBP has been confirmed by careful measurement of the results of 2-5 exercises.1 3 = CBP has been confirmed by careful measurement of the results of 5-10 exercises. 4 = CBP has been confirmed by careful measurement of the results of >10 exercises. 5 = CBP has been confirmed by a controlled, rigorously conducted scientific experiment or research effort specifically designed to test the CBP. Context Rating: 0 = CBP has not been demonstrated or discussed in any Red Team context. 1 = CBP has been used / tested / validated / asserted to apply in a single Red Team context. 2 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in two Red Team contexts. 3 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in more than two Red Team contexts, but not all contexts. 4 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in all or almost all Red Team contexts. Consensus Rating: N/A = There is only a single source that discusses the CBP. 0 = There are clearly more sources that disagree with the CBP than agree with it. 1 = The number of sources that agree with the CBP is roughly the same as those that disagree with it. 2 = Most sources, but not all, agree with the CBP. 3 = All sources agree with the CBP. 1 Note that if there was no careful measurement that informed the confirmation of the CBP, then it cannot receive a score greater than 1. Peer reviewed publications can be assumed to involve careful measurement.

Upload: others

Post on 16-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice Supporting Materials

Ratings for Candidate Best Practices (CBPs)

Overall Rating:

* Weakly Supported – Widely proposed as a best practice, but little to no empirical evidence, or empirical evidence that is contradictory. ** Moderately Supported – Some, but not conclusive empirical evidence in favor of the best practice, or the scope of the best practice is limited to a narrow context, such as one type of Red Teaming only. *** Strongly Supported – Multiple streams of empirical evidence from a number of Red Teaming contexts argue in favor of the best practice. Validation Rating:

0 = No validation; CBP is based on speculation alone. 1 = CBP has been derived based on the conduct / outcome of a single simulation/exercise. 2 = CBP has been confirmed by careful measurement of the results of 2-5 exercises.1 3 = CBP has been confirmed by careful measurement of the results of 5-10 exercises. 4 = CBP has been confirmed by careful measurement of the results of >10 exercises. 5 = CBP has been confirmed by a controlled, rigorously conducted scientific experiment or research effort specifically designed to test the CBP. Context Rating:

0 = CBP has not been demonstrated or discussed in any Red Team context. 1 = CBP has been used / tested / validated / asserted to apply in a single Red Team context. 2 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in two Red Team contexts. 3 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in more than two Red Team contexts, but not all contexts. 4 = CBP has been used / tested / validated / asserted to apply or is highly likely to apply in all or almost all Red Team contexts. Consensus Rating:

N/A = There is only a single source that discusses the CBP. 0 = There are clearly more sources that disagree with the CBP than agree with it. 1 = The number of sources that agree with the CBP is roughly the same as those that disagree with it. 2 = Most sources, but not all, agree with the CBP. 3 = All sources agree with the CBP.

1 Note that if there was no careful measurement that informed the confirmation of the CBP, then it cannot receive a score greater than 1. Peer reviewed publications can be assumed to involve careful measurement.

Page 2: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #1

* A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful, a Red Team cannot be influenced by, or appear to be influenced by, parties or concerns outside of the simulation process.

Source Extracts:

Source Extract Page(s) Validation Context

Command Red Team, Joint Command (May 2016)

"(4) The command red team is an integral but independent part of the staff. Red teams require organizational and intellectual independence to minimize the influence of factors that might otherwise normalize their efforts."

1-5 None provided Other (Military

Planning)

Command Red Team, Joint Command (May 2016)

"Although red teams need to maintain a degree of intellectual and organizational independence, they must also be integrated into the staff’s regular processes. To facilitate operations, teams should be identified as a distinct entity on organizational charts, telephone listings, and command websites. In addition, the team should be included as a core member of those planning teams that address the command’s most pressing problem sets. "

vi None provided Other (Military

Planning)

Command Red Team, Joint Command (May 2016)

"The command red team is distinguished by organizational independence, which helps it view problems from more analytically independent perspectives..."

1-2 None provided Other (Military

Planning)

Mulvihill, Timothy. What Can Stifle a Red Team? Medium, Homeland Security (October 14, 2014)

"An additional factor that can result in successful or unsuccessful red teaming is independence from the organization being red teamed. If a red team is not independent of the agency that they are red teaming, their findings may not be impartial, and therefore, their recommendations less effective. If red teamers are analyzing the organizations they belong to, their role may be diminished to merely providing different ideas rather truly challenging assumptions."

N/A, Web None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"Red teaming is the independent application of a range of structured, creative and critical thinking techniques to assist the end user make a better informed decision or produce a more robust product. "

1-3 None provided. General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

1-3: "Red teaming is the independent application of a range of structured, creative and critical thinking techniques to assist the end user make a better informed decision or produce a more robust product. " Lexicon-4: "Independence: Meet the requirement to provide alternative views and avoid group-think by being separate from the planning or project process. "

1-3, Lexicon-4 None provided General

Page 3: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Sandoz, John F. Red Teaming: Shaping the Transformation Process, Annotated Briefing, Institute for Defense Analyses (2001)

"A major challenge in red teaming is maintaining Red’s independence from Blue. In a success-oriented culture, Red Teams are often viewed as useful only to the extent they prove the validity of Blue concepts. The BMDO Chop Shop, discussed earlier, produced a number of inexpensive, technically feasible counters to theater missile defense concepts. The Chop Shop was independent of the theater missile defense program office, reporting through BMDO’s Threat Countermeasures Division. But over time, the funding of Chop Shop’s countermeasure development dwindled as its results were perceived to threaten the theater missile defense program. Historically, some well-intentioned Red Team efforts eventually become co-opted by Blue unless senior management is committed to preserving Red’s independence from Blue."

10 Example case General

Sandoz, John F. Red Teaming: Shaping the Transformation Process, Annotated Briefing, Institute for Defense Analyses (2001)

"Second, Red Teams need independence from Blue programs and concepts. They do not function well as a form of peer review occurring under program manager supervision. However, they should be accountable to senior levels above program management"

11 None provided General

Context Rating: 4 Context Validity Rating:

General Red Teaming: 1 Other (Military Planning): 0

Consensus Rating: 3 Overall Assessment of Best Practice: There is consensus in the Red Teaming literature, with anecdotal evidence but no empirical validation.

Page 4: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #2

* Leadership must buy into the Red Teaming process. Without support from organizational leadership (especially direct supervisors), it becomes less likely that the resources necessary for appropriate Red Teaming activities will be sustained or that the results of the Red Teaming will shape organizational decisions or behavior.

Source Extracts:

Source Extract Page(s) Validation Context

Command Red Team, Joint Command (May 2016)

"Red teams cannot succeed without command support. Command support enables the team to challenge convention and the staff to appropriately weigh the team’s inputs."

vi None provided Other (Military

Planning)

Craig, Susan. Reflections from a Red Team Leader, Military Review, 60 (2007)

"But in order for a red team’s recommendations to be implemented, they need more than access; they need top engagement, or leadership that is committed to making changes based on red team findings."

60

Anecdotal, non-specific

(Personal experience)

General

Kardos, Monique and Patricia Dexter. A Simple Handbook for Non-Traditional Red Teaming, Australian Department of Defense, Joint & Operations Analysis Division, 26 (2017)

"Another key aspect of successful red teaming is buy-in from stakeholder(s) - regardless of the actual outcome of the activity. This means that stakeholders are committed to truly investigating the potential issues as part of the activity, and are prepared to accept the findings rather than expecting findings to match a preconceived and preferred result. It is another prerequisite for the success of red teaming activities."

26 Anecdotal,

non-specific General

Mulvihill, Timothy. What Can Stifle a Red Team? Medium, Homeland Security (October 14, 2014)

"Another important component of successful red teams is top cover. This is essentially the backing needed from senior leadership in order for red teams to operate effectively. In the absence of top cover, red teams may experience stonewalling, dismissal of suggestions, and other issues that result from a lack of support"

N/A, Web None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"To make sure we use red teaming successfully, the end user needs to: … develop a good working relationship, including regular contact with the red team leader, be accessible to the team leader throughout the task; engage with, and listens to, the red team; ensure the red team findings are heard, and acted on, by senior decision makers”

2-1 to 2-2 None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"Guidelines for good red teaming…3. Support the red team. Value and use its contribution to inform decisions and improve outcomes."

2-2 None provided Other(Military

Planning)

The Role and Status of DOD Red Teaming Activities, Department of Defense, (2003)

“Top cover is needed to ensure that the red team’s products not only have the requisite degree of independence, but are seriously considered as well (this does not imply acceptance). “

6 None provided General

Page 5: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Sandoz, John F. Red Teaming: Shaping the Transformation Process, Annotated Briefing, Institute for Defense Analyses (2001)

"If Red Teams are to contribute to transformation, they must have credibility with senior leadership. Some previous attempts at red teaming fell short because senior leadership either did not have confidence in the Red Team’s judgments or else was unwilling to implement changes implied from red teaming. Senior leadership must view Red Teams as a vital decision support tool. "

11 Anecdotal, non-

specific General

Santarcangelo, Michael. Want to Be a Better Security Leader? Embrace Your Red Team, CSO (September 29, 2017)

"A few months ago, I talked with Mokady about the need to 'sprint before getting forced to scramble.' During that conversation, I asked if he saw a trend in the security leaders advancing their positions. Without hesitation, he explained that the leaders embracing red teams had a deeper understanding and were demonstrating more success."

N/A, Web Vague anecdotal Cyber

Penetration Testing

Context Rating: 4 Context Validity Rating:

General Red Teaming: 1 Other (Military Planning): 0 Cyber Penetration Testing: 0

Consensus Rating: 3 Overall Assessment of Best Practice: There is consensus in the Red Teaming literature, with

anecdotal evidence but no empirical validation. Most sources cite the 2003 Defense Science

Board report.

Page 6: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #3

** Almost always avoid groupthink when Red Teaming. Groupthink introduces powerful biases that skew simulation results and strenuous efforts should be made to avoid it. The sole exception is when simulating adversaries who tend to display groupthink in their own decision making, in which case intentionally introducing groupthink effects can increase the fidelity of the red teaming. Source Extracts:

Source Extract Page(s) Validation Context

Command Red Team, Joint Command (May 2016)

"Cohesive teams may unconsciously stifle dissent and subtly discourage alternative thinking. Helping counter the effects of these institutional influences, while simultaneously presenting alternative perspectives, requires a cautious, balanced approach by the red team. Common forms of institutional bias and influence include group think, tribal think, no think, boss think, silent think, false consensus, death by coordination, institutional inertia, hubris, and conflict of interest. "

vii None provided General

Command Red Team, Joint Command (May 2016)

"Cohesive teams may unconsciously stifle dissent and subtly discourage alternative thinking. Such dynamics are a direct consequence of the normal social dynamics and hierarchical structure of military organizations. Helping counter the effects of these institutional influences, while simultaneously presenting alternative perspectives, requires a cautious, balanced approach by the red team."

3-6 None provided General

Gladman, Brad. The 'Best Practices' of Red Teaming, Canada Command Operational Research and Analysis (2007)

"By following these and other suggested practices, both red team and planning team members can avoid the potentially disastrous groupthink behavior.”

7

Cites ~10 studies related to groupthink

generally

General

The Red Team Handbook, TRADOG G-2, 9 (2019)

"To combat such [groupthink] and support better decision making, Red Teamers practice Groupthink Mitigation. This act helps groups establish dynamics more conducive to the free flow and sharing of information and the generation of quality alternatives. "

53-54 None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"...Simultaneously, the red team should be alert to any bias or group-think that may affect their own findings."

3-4 None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

1-3: "Red teaming is the independent application of a range of structured, creative and critical thinking techniques to assist the end user make a better informed decision or produce a more robust product. " Lexicon-4: "Independence: Meet the requirement to provide alternative views and avoid group-think by being separate from the planning or project process. "

1-3, Lexicon-4

None provided General

Page 7: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Context Rating: 4 Context Validity Rating:

General Red Teaming: 0 Consensus Rating: 3 Overall Assessment of Best Practice: While the cognitive phenomenon of groupthink is well-

documented in the psychology literature, there is little direct evidence of its impact on Red

Teaming specifically. However, it is widely asserted in the Red Teaming domain that groupthink

is generally to be avoided, and indeed that Red Teaming can be a corrective to groupthink in

organizations.

Page 8: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #4

* Avoid mirror imaging bias. Mirror imaging (attributing one's own beliefs and thinking to the adversary) is detrimental to Red Teaming because it prevents Red Team members from considering the adversary's point-of-view and thus undermines a core aim of the Red Teaming approach.

Source Extracts:

Source Extract Page(s) Validation Context

Zhang, Liza and Gigi Gronvall. Red Teaming the Biological Sciences for Deliberate Threats, Terrorism and Political Violence (2018)

"The military sees value in red team methodologies in order to counter “mirror imaging,” that is, blind spots that result from a bias that an adversary will share the same values, perceptions, and rationales." [Reference 1] [Reference 1]: Command NWD, “Maritime Commander’s Red Team Handbook,” 2011. http://www.act.nato.int/images/stories/events/2011/cde/rr_navyhandbook.pdf.

2 None provided General

Zhang, Liza and Gigi Gronvall. Red Teaming the Biological Sciences for Deliberate Threats, Terrorism and Political Violence (2018)

"The red team framework was primarily developed in response to the inherent weaknesses of traditional threat assessments, including “mirror imaging,” with the intention of incorporating an adversarial interpretation of the existing landscape and possible responses. Red teaming as a practice has been very effective at identifying hidden assumptions, incorporating multinational considerations, and avoiding confirmation bias. By tasking individuals to use whatever stratagems they can to defeat current defenses, and rewarding red team members for being successful, the problem of mirror imaging may be greatly reduced. We present in our paper a discussion of the particular red team framework utilized by the UK Ministry of Defense and its applications in addressing biological threats."

18 None provided General

Maritime Commander's Red Team Handbook, Navy Warfare Development Command (2011)

"Many traps can derail the planning process. Red teams assist the commander in critically examining the group’s planning and decision making to avoid many of these critical thinking traps. If it is to be effective at all stages of the planning process, the red team should be alert to the challenges outlined below: [Reference 1] Mirror imaging—applying own attitudes (values, beliefs, cultural concepts, capabilities, etc.) to others, thus gaining a flawed understanding of consequences and outcomes." [Reference 1] [Reference 1]: United Kingdom Development, Concepts, and Doctrine Centre, A Guide to Red Teaming, February 2010, pp. 29-30.

19 None provided General

The Role and Status of DOD Red Teaming Activities, Department of Defense, (2003)

"There are additional challenges for red teaming that provide surrogate adversaries including: Not capturing the culture of the adversary/competitor (but instead mirror images)"

4 None provided General

Page 9: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Red Teaming Tactics, Techniques, and Procedures, TRADOC G-2 (2015)

"The Cultural Perception Framework is used to assist Red Team members in apperceiving another culture. It complements the 9-Step Cultural Methodology by posing questions of much greater detail. In order to avoid mirror-imaging, its steps lead red teamers through a process of discovering another culture based on its underlying tendencies, habits, values and beliefs. It provides red teams an ability to consider the kinds of questions that must be asked of Subject Matter Experts, in order to provide alternative perspectives about that culture."

1 None provided General

The Applied Critical Thinking Handbook, TRADOC G-2 (2015)

"UFMCS’ curriculum revolves around some fundamental questions: How can cultural anthropology help me think about another culture without resorting to mirror imaging?"

5 None provided General

The Applied Critical Thinking Handbook, TRADOC G-2 (2015)

"Like Devil’s Advocacy and Team A/Team B techniques, red teaming analysis is aimed at freeing the analyst from the prison of a well-developed mind-set; in this case, the analyst’s own sense of rationality, cultural norms, and personal values"

174 None provided General

The Applied Critical Thinking Handbook, TRADOC G-2 (2015)

"Planning challenges: Many of the variables above manifest themselves in dysfunctional group behaviors that lead to poor planning and decision making outcomes. ... These challenges include: ...b. Mirror Imaging: Richards J. Heuer, who spent 45 years in the CIA working in collection operations, counterintelligence, intelligence analysis, and personnel security, views mirror imaging as “…filling gaps in the analyst’s own knowledge by assuming that the other side is likely to act in a certain way because that is how [your country or organization] would act under similar circumstances.”12 Mirror imaging occurs when you apply your attitudes about trends, capabilities, beliefs, culture onto another. Many American policy makers and analysts fell into this trap during the planning for Operation Iraqi Freedom (OIF)."

96-97 None provided General

The Applied Critical Thinking Handbook, TRADOC G-2 (2015)

"JP 2-0, Joint Intelligence states, “Red teams assist joint operation planning by validating assumptions about the adversary, participating in the war-gaming of friendly and adversary COAs, and providing a check on the natural tendency of friendly forces to “mirror image” the adversary (i.e., to ascribe to an adversary the same motives, intent, and procedures that guide friendly forces)."

216 None provided General

Joint Publication 2-0: Joint Intelligence, Joint Chiefs of Staff (2013)

"In particular, intelligence should recognize each adversary as unique, and avoid mirror imaging while realizing the possible bias involved in their assessment type. For example, current intelligence and warning intelligence estimates may assess the same indicators differently. Red teams can be used to check analytical judgments by ensuring assumptions about the adversary are sound and intelligence assessments help minimize mirror imaging and cultural bias."

56 None provided General

Sandoz, John F. Red Teaming: Shaping the Transformation Process, Annotated Briefing, Institute for Defense Analyses (2001)

"How Red Teaming Can Fail... Red Team failed to reflect cultural bias of real adversaries… The adaptive nature of adversaries is influenced by a variety of cultural, religious, and ethnic considerations. Failing to reflect those factors in red teaming risks understating the cultural asymmetries of future threats."

10 None provided General

Page 10: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Longbine, David. Red Teaming: Past and Present, United States Army School of Advanced Military Studies (2008)

"The U.S. Army defines red teaming as “a function executed by trained, educated, and practiced team members that provides commanders an independent capability to fully explore alternatives in plans, operations, concepts, organizations, and capabilities in the context of the operational environment and from the perspectives of our partners, adversaries, and others.” The key aspects of this definition are its emphasis on independent thinking, challenging the thinking inside the organization, incorporating alternative perspectives in an attempt to eliminate cultural and ethnocentric bias, and a focus on fully exploring alternatives."

7 None provided General

Landry, Thomas. Embracing the Devil: An Analysis of the Formal Adoption of Red Teaming in the Security Planning for Major Events, Naval Postgraduate School (2017)

"Implicit assumptions are more difficult to deal with and cause more problems for decision makers. Often, decision makers are unaware of their implicit assumptions. They are the result of mirror imaging, ethnocentrism, paradigm blindness, faith in trends, cultural contempt, and biases. These implicit assumptions skew the decision maker’s understanding of the operational environment, the problem, and potential solutions. Challenging the conscious or unconscious thinking behind these assumptions first alerts the decision maker that they exist, and then enables him to assess how they skew his understanding. Incorporating alternative analysis to identify and challenge these implicit assumptions facilitates a better understanding of the operational environment and enables the decision maker to construct a more accurate mental model of the system."

9 None provided General

Gladman, Brad. The 'Best Practices' of Red Teaming, Canada Command Operational Research and Analysis (2007)

"An equally common and dangerous tendency, which can be avoided through robust red teaming, is 'mirror-imaging'. This refers to the common tendency to fill "gaps in the analysts own knowledge by assuming that the other side is likely to act in a certain way" because that is how the analyst would act if the positions were reversed."

7 None provided General

Context Rating: 4 Context Validity Rating:

General Red Teaming: 0 Consensus Rating: 3 Overall Assessment of Best Practice: Although there are no direct empirical tests in the Red

Teaming literature, this appears to be universally accepted among the Red Teaming community

as an essential best practice.

Page 11: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #5

* Red Teams generally benefit from diversity. In almost all cases, Red Teaming is more efficient if there is diversity among Red Team members in terms of their knowledge, experience, demographics and/or cultural background. This allows for a broader range of perspectives, more multi-faceted analysis and more synergistic collaboration amongst team

members.

Source Extracts:

Source Extract Page(s) Validation Context

Domestic Nuclear Detection Red-Teaming, Nobilis, Inc. 2007

"Accurate emulation of the threat actors is critically important in the proposed new red-team approach that is based on a more in-depth understanding of the terrorism methods, means, and tactics combined with a commensurate level of technical expertise for the desired capability level of the red team. This will be achieved by the following combination: Culturally and Educationally Diverse Team: The Red-Team composition will be engineered to include a broad perspective of views from a team with diverse cultural, international, educational, and experiential backgrounds."

7 None provided General

Kardos, Monique and Patricia Dexter. A Simple Handbook for Non-Traditional Red Teaming, Australian Department of Defense, Joint & Operations Analysis Division, 26 (2017)

"Ideally, critical thinking is coupled with other techniques to sample broader perspectives; alternatively, choosing a broad participant base (in terms of knowledge, skills and backgrounds) provides robust insights from multiple lenses and viewpoints."

36 None provided General

Ham, Carter et. al. Red Team Reign: Red Team Support to Joint Task Force Decision Processes, Red Team Journal (2010)

"Because of its broad interdisciplinary approach, red teaming works best when there is a wide array of experience among its members. Collaboration among team members with expertise in diverse areas is imperative for a red team to have maximum effectiveness. A red team must be just that—a team; a small group of people from different backgrounds who collectively study a people or region and are able to work together to form an overall impression of the enemy’s motives and actions."

6 None provided General

Sandoz, John F. Red Teaming: Shaping the Transformation Process, Annotated Briefing, Institute for Defense Analyses (2001)

"The Red Team needs people with diverse backgrounds such as cover, concealment and deception, weapons of mass destruction, information operations, and special operations. Part-time subject matter experts from a broad range of government, academic, and industry partnering organizations could be drawn in as needed to augment the Red Team. Expanding this base to eventually include foreign military representatives could bring a coalition perspective to strategy and concept development."

19 None provided General

The Red Team Handbook, TRADOC G-2, 9 (2019)

"Teams, especially when comprising members with diverse areas of expertise and perspectives, can better anticipate how the implementation of the new idea might impact other aspects of the organization."

63 None provided General

Page 12: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"The skill of the red team leader is in accommodating a diverse set of individuals who may hold novel views and making sure that they produce meaningful outputs."

2-4 None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"A red team should be tailored appropriately for the project being considered. Critical and creative thinkers will form the core of the team. The wider team may comprise a diverse mix of skills and experience or may be focused in one particular area, depending upon the issue being addressed. There is a need to include experts, but there must also be room for people who are able, and unafraid to ask, naïve questions. The team may be formed from a wide range of sources including MOD staff, Dstl4 analysts, partners across government, subject matter experts, academia, industry and commerce, think tanks, diaspora and international partners."

2-5 None provided General

Craig, Susan. Reflections from a Red Team Leader, Military Review, 60 (2007)

"Diversity in red team composition is very important. The value of diversity—in rank, service, expertise, age, and gender—was evident just by our class composition, which included not only officers and warrant officers from the Army’s Active and Reserve Components, the Marine Corps, and the Navy, but also civilians. Our varied experiences, perspectives, and expertise fostered dynamic classroom dialog and debate."

60 None provided General

Red Teaming of Advanced Information Assurance Concepts, Sandia National Laboratories (n.d.)

"Red teams are formed with the assumption that diverse teams deliver the best results...Additional expertise is sought, depending on the system under consideration. For example, a red team examining a biological and chemical agent detection system could include experts on biological and chemical warfare agents, physicians, and even meteorologists. A red team that evaluates a petroleum storage system might include petroleum engineers as well as information systems analysts."

2 None provided Cyber

Penetration Testing

Context Rating: 3 Context Validity Rating:

General Red Teaming: 0 Cyber Penetration Testing: 0

Consensus Rating: 3 Overall Assessment of Best Practice: There is consensus in the Red Teaming literature, with

anecdotal evidence but no empirical validation.

Page 13: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

Best Practice #6

* A Red Team needs a clear mandate. Successful Red Teaming begins with defining the scope and objectives of the team. This helps to ensure that the team remains focused on the intended issues, assists with creating benchmarks for evaluating the team's

performance, and increases the likelihood of its outputs being accepted and implemented.

Source Extracts:

Source Extract Page(s) Validation Context

Kardos, Monique and Patricia Dexter. A Simple Handbook for Non-Traditional Red Teaming, Australian Department of Defense, Joint & Operations Analysis Division, 26 (2017)

"Ideally, the design of every activity is driven by the aim and objectives – this ensures that resources (funds, personnel hours, consumables, etc.) are expended in the way most appropriate for achieving the outcome. This is particularly important in any context where demonstrating value for money is required in order to justify spending ever shrinking budgets on such activities."

38 None provided General

Bartels, Elizabeth. Building a Pipeline of Wargaming Talent: A Two-Track Solution, War on the Rocks (November 14, 2018)

"The success of any red team effort is contingent on a clear understanding of its mandate, and its expected outputs. This must be derived from a clear statement of the commander's intent, which will provide the requisite focus for the red team efforts, assist in its composition, and allow a measure of its success or failure. [Reference 1] This is arguably one of the most important elements in the success of any red team effort, and one which will guard against wasted effort and a lack of focus. Not surprisingly, experience seems to suggest that without clear and rational direction and specific deliverables, red teams tend to fail." [Reference 1]: Anna M. Culpepper "Effectiveness Of Using Red Teams To Identify Maritime Security Vulnerabilities To Terrorist Attack," p. 11.

18 None provided General

Gladman, Brad. The 'Best Practices' of Red Teaming, Canada Command Operational Research and Analysis (2007)

"Critical to the success of any model, game, or simulation is the stated objective. There are many reasons to conduct an exercise such as education, technical or doctrinal evaluation, research, or planning. However, in order to evaluate its effectiveness or mark the team’s progress, the hosting enterprise must state the purpose(s) at the outset. (Brewer & Shubik, 1979)"

11 None provided General

Red Teaming Guide Second Edition, UK Ministry of Defense, 2 (January 2013)

"To make sure we use red teaming successfully, the end user needs to... give clear objectives, defining the scope, timeframe and delivery mechanism for the task."

16 None provided General

Zenko, Micah. Red Team: How to Succeed by Thinking Like the Enemy, New York: Basic Books. (2015)

"Correctly framing the scope of activities that the red team will undertake is just as crucial as its structure, though this is often underappreciated. Before starting its work, the red team must have an explicit and mutual understanding with the targeted institution about exactly who or what is to be red teamed, for

8 None provided General

Page 14: Best Practice Supporting Materials...Best Practice #1 * A Red Team must be able to operate independently. For the results of its activities to be credible and analytically useful,

how long, with what degree of flexibility, and to what ultimate end."

Context Rating: 3 Context Validity Rating:

General Red Teaming: 0 Consensus Rating: 3 Overall Assessment of Best Practice: There is consensus in the Red Teaming literature,

although there has been no empirical validation.