judgment tech emanual - hogan assessments

JudgmentAssessment Technical Manual

Hogan Judgment Assessment Technical Manual

Hogan Assessment Systems Tulsa, OK 74114, USA

2014

© 2014 Hogan Assessment Systems, Inc.

No part of this work may be copied or transferred to any other form or expression without the expressed written consent of Hogan Assessment Systems, Inc.

Hogan Personality Inventory ™

Hogan Development Survey ™

Motives, Values, Preferences Inventory ™

Are exclusive registered trademarks of Hogan Assessment Systems, Inc.

hoganassessments.com

978-0-9889286-9-5

CONTENTS

III

CONTENTS 1. Conceptual Background 1

1.1 Introduction 1

1.2 Heuristics, Biases, and Rationalization 1

1.3 Judgment and Leadership 3

1.4 Current State of Judgment and Decision-Making Research 4

1.5 The Hogan Judgment Model 5

1.5.1 Information Processing 5

1.5.2 Using Personality to Assess Decision-Making Approaches 6

1.5.2.1 Threat Avoidance vs. Reward Seeking 7

1.5.2.2 Tactical vs. Strategic Thinking 7

1.5.2.3 Data-Driven vs. Intuit ive Decisions 8

1.5.3 Using Personality to Assess Reactions to Feedback 8

1.5.3.1 Defensive vs. Cool-Headed 9

1.5.3.2 Denial vs. Acceptance 9

1.5.3.3 Superficial vs. Genuine Engagement 10

2. Inventory Construction, Reliabil ity, and Structural Psychometrics 11

2.1 Development 11

2.1.1 Information Processing 11

2.1.1.1 Numerical Information Processing 12

2.1.1.2 Verbal Information Processing 12

2.1.2 Decision-Making Approaches & Decision-Making Styles 13

2.1.3 Reactions to Feedback & Openness to Feedback and Coaching 13

2.2 Definit ions of the Scales 14

2.3 Item Composition of the Scales 16

CONTENTS

IV

2.3.1 Numerical Information Processing 16

2.3.2 Verbal Information Processing 16

2.3.3 Decision-Making Approaches and Reactions to Feedback 16

2.4 Descriptive Statistics and Reliabil it ies 17

2.5 Parallel Forms Evidence for Numerical and Verbal Scales 19

2.6 Intercorrelations Between Hogan Judgment Model Scales 20

2.7 Structural Psychometrics of the Pre- and Post-Decision Scales 21

2.7.1 Pre-Decision Scales 22

2.7.2 Post-Decision Scales 23

3. Validity 24

3.1 Construct Validity 24

3.1.1 Procedure 24

3.1.2 Samples and Instruments 24

3.1.3 Results of Scale to Scale Correlates 31

3.2 Correlations with Others’ Descriptions 40

3.2.1 Procedure 40

3.2.2 Samples and Instruments 40

3.2.3 Results of Scale and Observer Description Correlates 41

3.3 Criterion-Related Validity 46

3.3.1 Procedure 46

3.3.2 Sample and Instrument 46

4. Interpretation and Uses 48

4.1 Information Processing 48

4.1.1 Verbal Information Processing 48

4.1.2 Numerical Information Processing 49

CONTENTS

V

4.1.3 Information Processing Styles 49

4.1.3.1 Deliberate 49

4.1.3.2 Quantitative 50

4.1.3.3 Qualitative 50

4.1.3.4 Versati le 50

4.2 Decision-Making Approaches 50

4.2.1 Threat Avoidance vs. Reward Seeking 51

4.2.1.1 Developmental Recommendations for Threat Avoidance 52

4.2.1.2 Developmental Recommendations for Reward Seeking 52

4.2.2 Tactical vs. Strategic Thinking 53

4.2.2.1 Developmental Recommendations for Tactical Thinkers 54

4.2.2.2 Developmental Recommendations for Strategic Thinkers 54

4.2.3 Data-Driven vs. Intuit ive Decisions 55

4.2.3.1 Developmental Recommendations for Data-Driven Decisions 56

4.2.3.2 Developmental Recommendations for Intuitive Decisions 56

4.3 Decision-Making Styles 57

4.3.1 Auditors 57

4.3.2 Surgeons 58

4.3.3 Stock Traders 58

4.3.4 Defense Analysts 58

4.3.5 Polit icians 58

4.3.6 Chess Players 58

4.3.7 Promoters 58

4.3.8 Investors 59

4.4 Reactions to Feedback 59

CONTENTS

VI

4.4.1 Defensive vs. Cool-Headed 59

4.4.2 Denial vs. Acceptance 60

4.4.3 Superficial vs. Genuine Engagement 60

4.5 Feedback Response Patterns 60

4.5.1 Defensive, Denial, Superficial 61

4.5.2 Defensive, Acceptance, Superficial 61

4.5.3 Defensive, Denial, Genuine 61

4.5.4 Defensive, Acceptance, Genuine 61

4.5.5 Cool-Headed, Denial, Superficial 61

4.5.6 Cool-Headed, Denial, Genuine 62

4.5.7 Cool-Headed, Acceptance, Superficial 62

4.5.8 Cool-Headed, Acceptance, Genuine 62

4.6 Openness to Feedback and Coaching 62

4.6.1 Resistant 63

4.6.2 Neutral 63

4.6.3 Receptive 63

5. Administering the Hogan Judgment Report 65

5.1 Key Features of the Online Assessment 65

5.2 Completing the Assessment Using the Online Platform 65

5.2.1 Numerical Section 68

5.2.2 Verbal Section 73

5.2.3 Decision-Making Style Section 78

5.3 Participant Informed Consent 82

5.3.1 Consent 83

5.3.2 Purpose 83

CONTENTS

VII

5.3.3 Data Processing 83

5.3.4 Access to Data 83

5.3.5 Security 84

5.3.6 Contact 84

5.4 Accommodating Individuals with Disabil it ies 84

5.5 Frequently Asked Questions 84

REFERENCES 86

APPENDIX A: Sample Hogan Judgment Report 98

APPENDIX B: Complete Correlation Matrices 107

TABLES & FIGURES

VIII

TABLES & FIGURES

Figure 1.1 Hogan Judgment Model ................................................................................... 5

Table 2.1 Classical Scale Statistics for Hogan Judgment Model Scales ......... 18

Table 2.2 Parallel Forms Evidence for Numerical Information Processing ..... 19

Table 2.3 Parallel Forms Evidence for Verbal Information Processing ............ 20

Table 2.4 Correlations between Hogan Judgment Model Scales ........................ 20

Table 2.5 Sample Ethnic Composition by Age and Gender ................................... 21

Table 2.6 Varimax Rotated Factor Matrix for Pre-Decision Scales .................... 22

Table 2.7 Varimax Rotated Factor Matrix for Post-Decision Scales .................. 23

Table 3.1 Construct Validity Evidence for Threat Avoidance vs. Reward

Seeking Scale .......................................................................................................................... 31

Table 3.2 Construct Validity Evidence for Tactical vs. Strategic Thinking

Scale ............................................................................................................................................ 33

Table 3.3 Construct Validity Evidence for Data-Driven vs. Intuit ive Decisions

Scale ............................................................................................................................................ 34

Table 3.4 Construct Validity Evidence for Defensive vs. Cool-Headed Scale . 35

Table 3.5 Construct Validity Evidence for Denial vs. Acceptance Scale .......... 36

Table 3.6 Construct Validity Evidence for Superficial vs. Genuine

Engagement Scale .................................................................................................................. 37

Table 3.7 Construct Validity Evidence for Openness to Feedback and

Coaching Scale ........................................................................................................................ 38

TABLES & FIGURES

IX

Table 3.8 Construct Validity Evidence for Verbal Information Processing

Scale ............................................................................................................................................ 39

Table 3.9 Construct Validity Evidence for Numerical Information Processing

Scale ............................................................................................................................................ 39

Table 3.10 Correlations with Threat Avoidance vs. Reward Seeking and

Observer Ratings .................................................................................................................... 41

Table 3.11 Correlations with Tactical vs. Strategic Thinking and Observer

Ratings ....................................................................................................................................... 42

Table 3.12 Correlations with Data-Driven vs. Intuit ive Decisions and


Table 3.13 Correlations with Defensive vs. Cool-Headed Reactions and


Table 3.14 Correlations with Denial vs. Acceptance and Observer Ratings ... 43

Table 3.15 Correlations with Superficial vs. Genuine Engagement and


Table 3.16 Correlations with Openness to Feedback & Coaching and


Table 3.17 Correlations with Verbal Information Processing and Observer

Ratings ....................................................................................................................................... 45

Table 3.18 Correlations with Numerical Information Processing and Observer

Ratings ....................................................................................................................................... 45

Figure 4.1 Information Processing Styles .................................................................... 49

Figure 4.2 Decision-Making Styles ................................................................................. 57

Figure 5.1 Participant Login Web Page ........................................................................ 66

TABLES & FIGURES

X

Figure 5.2 Participant Information Web Page ............................................................ 67

Figure 5.3 Participant Menu Wed Page ........................................................................ 68

Figure 5.4 Numerical Section Instructions .................................................................. 69

Figure 5.5 Numerical Section Sample Item ................................................................ 70

Figure 5.6 Numerical Section Sample Page ................................................................ 71

Figure 5.7 Numerical Section Review Page ................................................................ 72

Figure 5.8 Out of Time Page ............................................................................................. 73

Figure 5.9 Verbal Section Instructions ......................................................................... 74

Figure 5.10 Verbal Section Sample Item ..................................................................... 75

Figure 5.11 Verbal Section Sample Page .................................................................... 76

Figure 5.12 Verbal Section Review Page ..................................................................... 77

Figure 5.13 Decision-Making Style Section Instructions ....................................... 78

Figure 5.14 Decision-Making Style Section Sample Item ...................................... 79

Figure 5.15 Decision-Making Style Section Sample Page ..................................... 81

Figure 5.16 Decision-Making Style Section Review Page ...................................... 82

Table B.1 Correlations Between Judgment Scales and HPI Scales &

Subscales ............................................................................................................................... 107

Table B.2 Correlations Between Judgment Scales and HDS Scales &

Subscales ............................................................................................................................... 108

Table B.3 Correlations Between Judgment Scales and CPI Scales ................. 109

TABLES & FIGURES

XI

Table B.4 Correlations Between Judgment Scales and NEO PI-R Scales &

Facets ...................................................................................................................................... 110

Table B.5 Correlations Between Judgment Scales and IPIP Big 5 20-Item

Scales ...................................................................................................................................... 110

Table B.6 Correlations Between Judgment Scales and 16PF Scales ............. 111

Table B.7 Correlations Between Judgment Scales and TCI Scales ................. 111

Table B.8 Correlations Between Judgment Scales and MPQ Scales ............... 112

Table B.9 Correlations Between Judgment Scales and 6FPQ Factors & Facets

.................................................................................................................................................... 112

Table B.10 Correlations Between Judgment Scales and PRS Scales ............. 113

Table B.11 Correlations Between Judgment Scales and MVPI Scales &

Themes .................................................................................................................................... 114

Table B.12 Correlations Between Judgment Scales and CISS Interest & Skil l

Scales ...................................................................................................................................... 115

Table B.13 Correlations Between Judgment Scales and JPI-R Scales ........... 116

Table B.14 Correlations Between Judgment Scales and HBRI Scales ........... 116

Table B.15 Correlations Between Judgment Scales and SPA Cognitive

Abil it ies ................................................................................................................................... 116

Table B.16 Correlations Between Judgment Scales and BRI Cognitive Content

.................................................................................................................................................... 117

Table B.17 Judgment Scale Correlations with Observer Adjective Ratings . 118

Table B.18 Judgment Scale Correlations with Observer Ratings on

Descriptive Phrases ............................................................................................................ 119

1. CONCEPTUAL BACKGROUND

1

1. Conceptual Background 1.1 Introduction Queen Elizabeth I’s move to fight the Spanish Armada, Lincoln’s emancipation of the slaves, and Gandhi’s choice of nonviolent revolution are historical examples of decisions with positive outcomes (Axelrod, 2006). Conversely, Hannibal’s invasion of Rome, the Trojans’ acceptance of a gift horse, and General Custer’s tactical moves at Little Bighorn are decisions that led to disastrous consequences (Weir, 2005). The business world also provides many examples. Executives at Apple, Johnson & Johnson, 3M, and Intel made decisions that transformed their brands and boosted performance (Harnish, 2012), but decisions by leaders at Blackberry, Motorola, Lehman Brothers, Kmart, and Eastman Kodak cost their companies billions of dollars (McIntyre, 2012). Although the specific content and context of decision-making differ, the consequences of good or bad judgment are consistent. Considering the importance of good judgment, Hogan Assessment Systems developed the Hogan Judgment Report to help organizations understand the decision-making styles of their key employees. This manual describes: (a) the dynamics of human judgment; (b) the importance of judgment and decision-making for leader performance; (c) a multi-component model of judgment, (d) reliability and validity evidence for scales measuring components of the model; and (e) interpretation, uses, and administration of the Hogan Judgment Report. The research conducted to develop and validate the Hogan Judgment Model conforms to standards outlined in the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission [EEOC], 1978; hereafter “Uniform Guidelines”), the Principles for the Validation and Use of Personnel Selection Procedures (Society for Industrial and Organizational Psychology [SIOP], 2003; hereafter “Principles”), and the Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 1999; hereafter “Standards”). In areas where the Uniform Guidelines, Principles, or Standards proved vague or inapplicable, we relied on the broader scientific and professional literature for guidance. 1.2 Heuristics, Biases, and Rationalization People use a variety of methods to make decisions: logical reasoning; reviewing available data and background information; or applying heuristics (Gigerenzer & Gaissmaier, 2011). Logic- and data-based judgments are cognitively demanding, requiring knowledge of alternatives and their consequences and probabilities (Simon, 1979). In contrast, heuristics are mental shortcuts that require less cognitive effort (Shah & Oppenheimer, 2008). Historically, researchers have linked using logic and data with accurate judgment and using heuristics with error-prone judgment (Kahneman 2003; Tversky & Kahneman 1974). However, increasing awareness of the complexity of real-world decision-making contexts suggests that rational and data-driven reasoning are often impractical and may not guarantee effective outcomes (Busenitz & Barney, 1997; Gigerenzer & Gaissmaier, 2011).


2

According to Gigerenzer and Gaissmaier (2011), a heuristic is “a strategy that ignores part of the information, with the goal of making decisions more quickly, frugally, and/or accurately than more complex methods” (p. 454). Thus, people often balance speed against decision accuracy (Wilson, 2010). Heuristics are most useful when the cost of cognitive effort outweighs the benefits of accuracy (Payne, et al., 1993; Pachur & Hertwig, 2006; Shah & Oppenheimer, 2008). Examples of common heuristics include: • Affect – The affect heuristic is an emotion-based shortcut that relies on positive or

negative feelings attached to people, objects, or situations (Finucane, Alhakami, Slovic, & Johnson, 2000; Spence & Townsend, 2008; Zajonc, 1980). For example, if an individual dislikes strangers, he/she may decide to avoid them regularly.

• Availability – The availability heuristic involves estimating the likelihood of an event happening based on how easily one can recall similar events (Kliger & Kudryavtsev, 2010; Tversky & Kahneman, 1974; Wilson, 2010). For example, people may assess the likelihood of having a heart attack by recalling the number of family members who have had heart attacks.

• Effort – The effort heuristic is often used to set the value of something based on the

effort expended in trying to obtain it (Kruger, Wirtz, Van Boven, & Altermatt, 2003). For example, students may believe they deserve higher grades if they put more effort into class projects.

Researchers speculate about the mechanisms people use to select heuristics. Rieskamp and Otto (2006) propose that people select heuristics based on past learning experience. Others (e.g., Snook, et al., 2004) suggest explicit teaching and imitation affect the selection of heuristics. We think people select heuristics based on their personalities. Regardless of the underlying mechanisms, however, personal preferences for certain heuristics contribute to individual differences in judgment and decision-making. Biases represent another major cognitive shortcut (Busenitz & Barney, 1997). Aronson, Wilson, and Akert (2010) suggest people distort information to fit preconceived notions and/or align current situations with previous experiences. As with heuristics, biases consume few cognitive resources. Examples of common biases include: • Anchoring and Adjustment – The anchoring and adjustment bias involves making an

estimate based on an initial “anchor” value (provided or self-generated) and then adjusting up or down until a final answer is reached (Tversky & Kahneman, 1974). This bias is common in sales, where a salesman may provide an unreasonably high price for an item and then adjust the price to something that seems more reasonable in comparison.

• Gambler’s Fallacy – The gambler’s fallacy involves believing that chance is a self-correcting process, so that an outcome in one direction will increase the likelihood of an outcome in the opposite direction (Tversky & Kahneman, 1974). For example, a person may believe a coin that has landed on “heads” five times in a row is likely to land on “tails” next because it is “due.”


3

• Escalation of Commitment – Escalation of commitment involves continuing a course of

action because of previous investments (Schmidt & Calantone, 2002). This explains why people often spend more at auctions than they intend. Once they make a bid, they mentally commit money to it. Committing more money in small increments may seem easier than backing away from the bidding process.

• Framing – The framing bias occurs when small changes are made in the way a problem is presented, leading a decision maker to pick a positively framed option over a negatively framed one (Hodgkinson, Maule, Bown, Pearman, & Glaister, 2002). For example, people are willing to spend more for ground beef when it is labeled “80% lean” than “20% fat” (Levin, et al., 2002).

Although heuristics and biases are adaptive in some circumstances, they can lead to poor decisions. One common reaction to bad decisions is rationalization. According to Aronson, Wilson, and Akert (2010), recognizing bad decisions challenges our positive self-perceptions and creates cognitive dissonance. We may try to reduce this dissonance by (a) changing our behavior, (b) changing our beliefs, or (c) adding new beliefs. In other words, we may try to correct our mistakes or we may rationalize our poor decision. As with heuristics and biases, there are individual differences in rationalization across people. Overall, the study of human judgment indicates that individuals (a) often make irrational decisions, relying on heuristics and biases; (b) show predictable individual differences in their decision-making tendencies; and (c) react differently to the cognitive dissonance created by feedback regarding poor decisions. Understanding individual differences in how individuals approach decisions and react to feedback about poor decisions is critical for leaders whose decisions impact their organizations. 1.3 Judgment and Leadership Leaders mostly make decisions about strategy, staffing, and capital investments. Thus, good judgment is an essential component of effective leadership at both macro and micro levels. At the macro level, good judgment drives organizational profitability. Using a decision experiment, De Neys, et al. (2011) concluded that people who are willing to accept uneven splits of gains with their partners achieve higher overall payoffs. A study of more than 1,000 major business investments showed that when organizations work to reduce biases in their decision-making processes, they achieve higher returns (Lovallo & Sibony, 2010). At the micro level, the leader’s decisions affect not only organizational performance, but also everyone associated with them (Safi & Burrell, 2007). Consider the decisions made regarding Hurricane Sandy, the coaching scandal at Penn State University, or the use of chemical weapons in Syria. In each case, the decisions had significant consequences – in some cases life and death – for larger populations. Similarly, Russ, et al. (1996) found that the decision-making style of sales managers impacted employee performance, satisfaction, and commitment.


4

To understand decision-making, Hogan developed a model that defines individual differences in good judgment in terms of information processing style, tendencies in approaching decisions, and reactions to feedback about bad decisions. This assessment model facilitates leadership development by building strategic self-awareness regarding how leaders process information, approach business decisions, and react to feedback regarding decision outcomes. 1.4 Current State of Judgment and Decision-Making Research Researchers have focused on classifying human judgment processes. In the emerging field of decision science, researchers combine Economics (e.g., Rubinstein, 1998), Cognitive Psychology (e.g., Tversky & Kahneman, 1974), and Neuroscience (e.g., Bechara, 2005; Brocas, 2012) to understand decision-making. However, in reality people are seldom able to follow step-by-step logic processes (Mintzberg & Westley, 2001). For example, rather than “thinking first” (i.e., making decisions based on facts), people may “see first” (i.e., make decisions based on insights) and “do first” (i.e., making decisions through experimentation). After observing mid-career managers from across the globe, Mintzbeg and Westley (2001) conclude that “thinking first” is only functional when the issues are clearly defined, the context is structured, and reliable data are available, whereas “seeing first” and “doing first” enable visioning and venturing under novel and ambiguous situations. Although rationalization may sometimes lead to more accurate decisions, real decision-makers always make decisions under time pressure in unstructured contexts. Thus, knowing the strengths and challenges of a leader’s decision-making style provides insights into leadership competence. In addition to ignoring individual differences in irrational decision-making tactics, previous research also overlooks individual differences in reactions to decision outcomes. Becoming aware of such reactions is critical for leadership performance. The complexity of global business increases the difficulty of decision-making, which necessarily increases the incidence of biased and/or ineffective decisions. For example, a study of 356 decisions made in medium to large organizations in the U.S. and Canada reveals that 50% of managers’ decisions fail (Nutt, 1999; 2004). Given the consequences of leader decisions and the high rate of wrong decisions, it is critical to explore ways to improve leader judgment and decision-making. Understanding decision-making preferences is necessary but insufficient for improving judgment. Kahneman et al. (2011) conclude that becoming aware of judgment biases only minimally improves decision quality. Similarly, Milkman, et al. (2009) suggest going beyond the description of judgment biases to improve decision-making. One way to do this is to understand how people react to feedback about incorrect decisions. According to Fischhoff (1982), the best way to overcome judgment bias is to combine feedback with coaching. Brett and Atwater (2001) find people with a learning goal orientation are particularly receptive to negative feedback, leading to a greater likelihood of performance improvement. Similarly, Smither, et al. (2005) find leaders who react positively to feedback are more likely to benefit from multi-source feedback. These findings suggest there are important individual difference in receptiveness to feedback and greater receptivity is associated with greater benefits.


5

Feedback is particularly important for leaders, because as job level increases, they are responsible for making higher stakes decisions, but are also less likely to receive constructive feedback (Gallo, 2012). Moreover, leaders who are uncomfortable offering criticism are also less interested than others in receiving feedback (Kouzes & Posner, 2014; Zenger & Folkman, 2014). The Dunning-Kruger effect (Kruger & Dunning, 1999) suggests that poor performers not only make bad decisions, but also overestimate their talent. This can lead to a vicious cycle, especially for leaders whose decisions are critical to organizational performance. 1.5 The Hogan Judgment Model To fill these gaps in existing research on judgment, we developed the Hogan Judgment Model presented in Figure 1.1. This model: (a) accounts for individual differences in how leaders process information, (b) defines individual differences in pre-decision tendencies, and (c) examines post-decision reactions to feedback. Thus, this model provides a comprehensive approach to assessing business judgment by focusing on all the factors consequential to leadership judgment in the global business environment. Below, we introduce each component of the model in detail. Figure 1.1 Hogan Judgment Model

1.5.1 Information Processing Information processing style concerns how individuals acquire and process information. Information processing shapes the content of business decisions (Citroen, 2011). Although leaders typically have adequate cognitive ability, they differ in terms of their information preferences. The Hogan Judgment Model focuses on two types of information: (a) numerical data, and (b) verbal information. Vocational interest research shows people can be sorted into two broad groups: scientists and humanists (Ackerman & Heggestad, 1997; Snow, 1956). Scientists enjoy analyzing and interpreting objective problems rooted in data,


6

whereas humanists enjoy creating, analyzing, and interpreting emotional experiences. The information processing styles of these two tribes are distinct. Scientists are more effective at processing numerical data and identifying patterns and rules, whereas humanists are more effective at processing verbal information and understanding and using words to create desired effects. Decision-makers tend to specialize in either numerical data or verbal information. Although people can process both types of information, many process one type of information more efficiently than the other. Understanding the information processing styles of decision-makers can help organizations diversify their leadership teams and ensure that leaders have the right sources of information to make effective decisions. Similarly, individual leaders can use this information to develop their judgment by playing to their natural strengths and developing attributes that don’t come as naturally. 1.5.2 Using Personality to Assess Decision-Making Approaches Personality predicts a range of organizational performance measures, including overall job performance (Barrick, Stewart, & Piotrowski, 2002; Dudley, Orvis, Lebiecki, & Cortina, 2006; J. Hogan & Holland, 2003), team performance (Peeters, Van Tuijl, Rutte, & Reymen, 2006), expatriate performance (Mol, Born, Willemsen, & Van Der Molen, 2005), Counterproductive Work Behaviors (CWB; Berry, Ones, & Sackett, 2007; Gruys & Sackett, 2003), Organizational Citizenship Behaviors (OCBs), altruism, job dedication, interpersonal facilitation, and generalized compliance (Borman, Penner, Allen, & Motowidlo, 2001; Dubley et al., 2006; Hurtz & Donovan, 2000; LePine, Erez, & Johnson, 2002; Organ & Ryan, 1995). Personality also predicts individual differences in judgment and decision-making. Soane et al. (2010) show that personality predicts risky choices in a wide variety of social situations. Similarly, higher impulsivity is associated with faulty decision-making (Davis, Patte, Tweed, & Curtis, 2007). Dewberry et al. (2013) conclude that Five-Factor Model (FFM) personality variables explain variance in decision-making competence beyond decision-making and cognitive styles. In particular, Extraversion is negatively associated with competent decision-making, whereas Emotional Stability is positively associated with decision-making. Examining the dark side of personality, Zhu and Chen (2014) suggest that narcissism is related to CEOs’ tendency to adopt the corporate strategies of other boards. Chen et al. (2014) also find that CEO overconfidence influences responses to corrective feedback (e.g., inputs from markets, customers, employees, and other stakeholders). Similarly, Tang et al. (2014) propose that CEO hubris negatively impacts corporate social responsibility. Consistent with this research, the Hogan Judgment Report uses personality measures to assess individual differences in decision-making. Research on individual differences in judgment and decision-making has typically focused on judgment biases and their impact on real-life outcomes (e.g., Bruine de Bruin, et al., 2007; Stanovich & West, 2000). Measures of judgment (e.g., Bruine de Bruin et al., 2007; Bruine de Bruin et al., 2012; Parker & Fischhoff, 2005) study generic biases that may not impact business decisions. In contrast, the Hogan Judgment Model concerns three


7

dimensions that relate to business decisions: (a) threat avoidance vs. reward seeking; (b) tactical vs. strategic thinking; and (c) data-driven vs. intuitive decisions. 1.5.2.1 Threat Avoidance vs. Reward Seeking All decisions entail potential threats and rewards. For example, when Marissa Mayer was appointed CEO of Yahoo and banned staff telecommuting, she thought that enhanced employee collaboration would outweigh the risk of degrading staff morale. Individuals differ in their tendencies to avoid threats or seek rewards. According to loss-aversion theory, people are generally prone to negativity bias (e.g., Baumeister et al., 2001; Rozin & Royzman, 2001), which means they pay more attention to potential threats than rewards when evaluating decision options (e.g., Benartzi & Thaler, 1995; Kliger & Levit, 2009; McGraw et al., 2010). People also vary in their risk perceptions. For example, entrepreneurs perceive risks as less threatening (Cooper, Woo, & Dunkelberger, 1988) and focus on potential future rewards. McGhee, et al. (2012) found that Extraversion, Openness, and (low) Conscientiousness predict risk-taking in preadolescents. Similarly, research with adults suggests that Extraversion, positive affectivity, and sensitivity to Behavior Activation Systems (BAS) are associated with reward seeking, whereas Neuroticism, negative affectivity, and sensitivity to Behavior Inhibition Systems (BIS) are associated with failure avoidance (Atkinson, 1957; Elliot & Thrash, 2002). Personality variables such as sensation seeking, impulsivity, and low self-control are also associated with risky behavioral choices (Zuckerman, 2007). Organizations often need decision-makers with different levels of threat avoidance or reward seeking. For example, start-up companies need employees who will make bold decisions to expand the company’s bottom line, whereas investment advisors should make decisions that minimize potential threats to their clients’ finances (Weber, et al., 2002). Individuals can also use information about their tendency to avoid threats or seek rewards to develop their decision-making skills. 1.5.2.2 Tactical vs. Strategic Thinking In approaching decisions, some people focus on immediate, practical issues, whereas others focus on longer term challenges and opportunities. Tactical thinkers tend to evaluate cost, implementation, and staffing problems to the exclusion of longer term contextual issues (Leonard & McAdam, 2002). Conversely, strategic thinkers tend to consider “big picture” trends, capabilities, and sustainable gains (Citroen, 2011), but may neglect important logistical matters. Sustained success and growth requires both tactical execution and strategic vision. For example, Apple’s success reflects both Steve Jobs’ vision and the tactical execution of his designs. According to Pellegrino (1996), certain personality characteristics predict strategic thinking. Kauer, et al. (2007) found that, in top management teams, flexibility, achievement motivation, networking skill, and action orientation impact agenda-setting, generating strategic alternatives, and speed of strategic decision-making. Similarly, O’Connell et al.


8

(2013) found that personality predicts preferences for long-term vs. short-term rewards when people make decisions for others (O'Connell, et al., 2013). Organizations can use information about the tactical and strategic thinking preferences of their employees to select, position, and develop key talent. Strategic thinkers are critical to large-scale direction setting, whereas tactical thinkers are needed to implement this direction and maintain established structures (Leonard & McAdam, 2002). Similarly, individuals can use strategic self-awareness of their tendencies towards tactical or strategic thinking to ensure they consider both short- and long-term goals. 1.5.2.3 Data-Driven vs. Intuit ive Decisions Some people prefer to use data and facts to make decisions, whereas others prefer using intuition and experience as their guide. According to Kahneman (2003), people’s decisions are affected by two interacting systems: System 1 (intuition) and System 2 (deliberative thinking). Intuitive decision-makers prefer System 1, which is fast, automatic, and effortless. McNaught (2013) found that experienced leaders often rely on intuition. In contrast, data-driven decision makers prefer System 2, which is slow, controlled, and effortful. People who prefer System 2 collect as much information as possible regarding relevant alternatives, potential consequences, and probabilities (Simon, 1979). Deliberate thinking is positively correlated with a need for structure, need for cognition, Conscientiousness, and perfectionism, whereas intuitive thinking is correlated with Extraversion, Agreeableness, and Openness to Experience (Betsch, 2008)—i.e., both are related to personality. These differences in style are not necessarily correlated with decision accuracy. Data-driven decisions may seem more accurate because they are based on evidence, but the complexity of the global business environment often limits the amount of information available to support optimal decision-making. Moreover, information may quickly become outdated, which challenges the accuracy of data-driven decisions (Gigerenzer & Gaissmaier, 2011). In contrast, experienced leaders believe the high quality of intuitive decisions is a function of their experience (McNaught, 2013), although the accuracy of such decisions also seems limited (Choudhry, Fletcher, & Soumerai, 2005; Swets, Dawes, & Monahan, 2000; Tetlock, 2005). Whether data-driven or intuitive decision-making is more accurate depends on many factors. Within any organization, some decisions require careful consideration, whereas others must be made quickly. Individuals can use information concerning their own style to understand when they are taking too long or jumping too quickly into decisions. 1.5.3 Using Personality to Assess Reactions to Feedback

After people make decisions, personality predicts their reactions to feedback about their decisions. Although past research demonstrates that performance can be improved with feedback (Ilgen, Fisher, & Taylor, 1979; Kluger & DeNisi, 1996; Latham & Locke, 2007; Salmoni, Schmidt, & Walter, 1984), meta-analytic research shows that feedback often reduces performance (Kluger & DeNisi, 1998), suggesting that there are important individual differences in reactions to feedback. A recent Harvard Business Review article (Zenger & Folkman, 2014) shows that among 2,500 people surveyed about reactions to


9

feedback, 52.0% found negative feedback helpful, whereas 47.5% preferred to receive positive feedback. Personality characteristics such as self-esteem and resiliency predict their acceptance of feedback (“How Employees Process,” 2013; Shrauger & Rosenberg, 1970). For example, individuals with low self-esteem perceive critical feedback as too harsh, which inhibits their task performance. In addition, Steers (1975) found that, for high achievement individuals, there is a positive relationship between the amount of feedback provided and subsequent performance. Consistent with this, the Hogan Judgment Report assesses individual differences in reactions to feedback about ineffective decisions. Feedback is critical for any successful developmental initiative, as shown by research on goal-setting and self-regulatory motivation (Button, Mathieu, & Zajac, 1996; Donovan & Williams, 2003; Locke & Latham, 1990). Individuals who react positively to feedback are more likely to set difficult and specific goals (Ilies & Judge, 2005; Ilies, Judge, & Wagner, 2010), which leads to increases in individual and organizational performance (e.g., Latham & Locke, 2007; Locke, 1997; Locke & Latham, 1990). Personality predicts how people respond to negative feedback. For example, Bell and Arthur (2008) found that Extraversion and Agreeableness positively affect feedback acceptance. Similarly, Smither, et al. (2005) found that leaders higher in Adjustment, Sociability, and Conscientiousness were more willing to accept feedback, seek additional feedback, and use feedback to improve performance. The Hogan Judgment Model characterizes reactions to feedback about ineffective decisions along three dimensions: (a) defensive vs. cool-headed; (b) denial vs. acceptance; and (c) superficial vs. genuine engagement. 1.5.3.1 Defensive vs. Cool-Headed Some individuals respond to negative feedback by becoming emotionally volatile and defensive, and blaming external factors beyond their control. Such individuals tend to be challenged in their ability to recognize how they have contributed to the mistakes and how they can improve their performance in the future. In contrast, individuals who remain calm after negative feedback are more likely to recognize personal faults and set goals for improvement. For example, individuals with an internal locus of control are more likely to accept feedback than individuals with an external locus of control because they take ownership of their behaviors and are motivated to make changes (Feather, 1968). 1.5.3.2 Denial vs. Acceptance According to Freud (1946), denial is a common defense people use to protect themselves from negative thoughts and feelings. Some people react to negative feedback by denying the data, whereas others more readily accept the criticism. Although denial can make people feel better, it does not help them make better decisions in the future. For example, Kluger and DeNisi (1998) found that individuals who take negative feedback personally are unlikely to improve their subsequent performance.


10

1.5.3.3 Superficial vs. Genuine Engagement Some people’s reactions to feedback are motivated by a desire to maintain positive social relations (Aitkenhead, 1984). According to Ashford and Northcraft (1992), the tendency to manage impressions often overrides listening to feedback in evaluative contexts. Research on collegiate swimmers found that those with high self-esteem find negative feedback motivating (Marsden, 1998). Similarly, business leaders who genuinely listen to negative feedback are more likely to make better decisions in the future than leaders who listen to feedback only to maintain positive social impressions. In general, the more calm, accepting, and engaged a person is when receiving negative feedback, the more coachable that person is, and the more likely they are to improve future decisions. That is, individuals who are receptive to coaching are more likely to recognize their mistakes, take responsibility for them, learn from them, and improve future decisions compared to individuals who are generally resistant to coaching. Furthermore, although all three characteristics relate to reactions to feedback, they are independent. To improve decision-making, one must consider his/her tendencies on all three dimensions and reflect on these tendencies when faced with negative feedback.

2. INVENTORY CONSTRUCTION, RELIABILITY, & STRUCTURAL PSYCHOMETRICS

11

2. Inventory Construction, Reliabil ity, and Structural Psychometrics

2.1 Development As noted in Chapter 1, the Hogan Judgment Model assumes that, although individual judgment and decision-making is biased and irrational, it is still predictable (Ariely, 2008). Although some cognitive shortcuts (i.e., heuristics; Gigerenzer & Gaissmaier, 2011; Shah & Oppenheimer, 2008) are advantageous, others (i.e., biases; Busenitz & Barney, 1997) cause people to distort information and make bad decisions. Moreover, individuals typically rationalize their biased decision-making after the fact; as a result, they internalize accurate decisions and blame failed decisions on external sources (Aronson, Wilson, & Akert, 2010). How leaders make decisions and respond to failure is especially consequential because they control resources (Lovallo & Sibony, 2010) and typically make bad decisions at least half of the time (Nutt, 1999; 2004). Researchers usually attempt to assess “good judgment” as the ability to make correct decisions. These efforts often focus on critical thinking, IQ, and other cognitive measures. However, this approach (a) ignores non-cognitive individual differences that affect decision-making, (b) provides no information for developing judgment, and (c) ignores the fact that highly intelligent people often make bad decisions. To understand this last issue, some researchers have studied how personality and values affect decision-making. However, few have studied the interaction between cognitive and non-cognitive attributes. In our view, people with good judgment tend to not only make good decisions, but also recognize their mistakes and take corrective action. Our assessment concerns individual differences in (a) information processing, (b) decision-making, and (c) reactions to feedback about failed decisions. 2.1.1 Information Processing For the cognitive component of the judgment assessment, we drew from research on learning potential and vocational interests. Learning potential concerns the ability to acquire knowledge and information (Hamers & Ruijssenaars, 1992). Some people learn more quickly than others. Faster learners have advantages in educational settings, and in complex, rapidly-changing work environments. Also, the more complex a job, the less incumbents can rely on previously acquired knowledge to solve problems. Thus, learning potential is consequential in complex jobs. Research on vocational interests shows that people can be sorted into two tribes: scientists and humanists (Snow, 1956; Ackerman & Heggestad, 1997). Scientists enjoy analyzing and interpreting data, whereas humanists enjoy creating, analyzing, and interpreting powerful emotional experiences. Science is about prediction and control, and the humanities give form to feeling. These tribes are also characterized by distinct numerical and verbal learning styles. Numerical learning potential concerns the ability to identify patterns and rules, which predicts interest in science and math and performance in fields such as finance, accounting, engineering, and IT. Verbal learning potential concerns the ability to understand and use


12

words to create desired effects, which predicts interest in literature, philosophy, journalism, and advertising and performance in fields requiring good communicational skills. Numerical and verbal potential are related, such that some individuals are slow and deliberate in processing both types of information and others are able to quickly and efficiently process both types. However, many individuals tend to prefer one type of information over the other. Based on these considerations, we included numerical and verbal elements in the cognitive component of the Hogan Judgment Model. We chose these not only because of their importance in the vocational interest literature, but also because they represent the sources of information most frequently used in making organizational decisions. That is, before making a decision, leaders typically refer to numerical data (e.g., sales, stock performance) and/or verbal information (e.g., emails, reports) to inform their decisions. We wrote two sets of parallel items for each section. 2.1.1.1 Numerical Information Processing For numerical information processing, we wrote items using multiple formats including balancing equations, identifying larger or smaller fractions, sequence completion, and pattern identification and completion. Although all items could be answered correctly by most professionals with unlimited time and resources, we imposed a 10-minute limit. This reflects the fact that real-world decision-makers have limited time to review data before making a decision. We began testing these items in the fall of 2011 using the Numerical Reasoning Test 20 (NRT-20; Chamorro-Premuzic, 2008) as a guide. We chose the NRT-20 as a marker because it does not require previous knowledge or specialized training in mathematics. We tested new items with research samples and revised items as needed to improve their psychometric properties. We also solicited input from experts and colleagues concerning item content and accuracy. We collected data through Hogan’s on-line platform. The system is fully redundant, using a multi-location systems architecture to ensure constant accuracy and availability. 2.1.1.2 Verbal Information Processing For verbal information processing, we wrote items requiring users to solve problems concerning the placement of letters in the alphabet, types of items, analogies, opposites, object size, and meanings. We wrote half using positive wording (e.g., “X is a type of Y”) and the other half using negative wording (e.g., “X is not a type of Y”). The items require test-takers to recognize patterns in verbal data and use that information to determine whether the item is True or False. We also imposed a two-minute time limit. This limit helps us better determine how quickly individuals can process verbal information, reflecting the fact that organizational decision-makers have limited time to make decisions. We began testing these items in the fall of 2012 using assessments designed to measure working memory (e.g., Baddeley, 1986; 1997) as a starting point. Next, using item types from other verbal assessments, we began testing items using convenience samples. Again,


13

we revised items as needed and solicited feedback from experts and colleagues. We collected data using the Hogan platform previously described. The Hogan Judgment Model describes information processing style using one of four categories: (a) deliberate, (b) qualitative, (c) quantitative, or (d) versatile. We define these categories in a later section of this chapter. 2.1.2 Decision-Making Approaches & Decision-Making Styles To assess decision-making approaches, we identified three independent dimensions of individual preferences in approaching decisions: (a) goal-orientation (i.e., threat avoidance vs. reward seeking), (b) thinking style (i.e., tactical vs. strategic), and (c) base of information (i.e., data-driven vs. intuitive). Next, we identified scales from these assessments that predicted behaviors associated with our overall three themes (e.g., MVPI Security and threat avoidance; HPI Ambition and strategic thinking; MVPI Scientific and data-driven decisions) and identified items from the scales that seemed to reflect each theme (e.g., “I don’t like unpredictable situations”, “I am a leader in my group”, “I am interested in science”, etc.). We also wrote items with work-related content for each construct. Specifically, we wrote items to reflect what a person with a given tendency is likely to do when approaching a work-related decision. We developed these scales with non-overlapping content and homogenous themes to avoid overlap. We began studying tactical and strategic thinking preferences in 2009, and began writing items for the remaining pre-decision scales in the fall of 2011. We developed each scale by combining existing assessment content with new items, testing the scale with research samples, analyzing the data to calculate statistical properties, and making revisions as needed to enhance the psychometric properties of each scale. We again solicited expert feedback and collected data using Hogan’s platform. We used the three scales previously described to define eight types, each represented by an occupation representing a combination of approaches to making business decisions. These occupational types are not indicative of likely vocational interests or occupational performance, but are intended to describe an individual’s general decision-making style. For example, a person characterized as reward-seeking, strategic, and data-driven would be classified as an “Investor” (e.g., Warren Buffett). Such individuals prefer to maximize their long-term rewards based on data-driven decisions. 2.1.3 Reactions to Feedback & Openness to Feedback and Coaching Our strategy for writing items to assess reactions to negative feedback paralleled the process for developing pre-decision scales, but focused on tendencies individuals use when reacting to feedback about failed decisions. Again, we started with three general themes: (a) initial response (i.e., defensive vs. cool-headed reactions), (b) feedback synthesis (i.e., denial vs. acceptance), and (c) ongoing engagement (i.e., superficial vs. genuine). The high end of each scale reflects more positive reactions to feedback, whereas the low end reflects


14

tendencies to “shoot the messenger”, “ignore the messenger”, and “mistrust the messenger”, respectively. Next, we identified specific scales predictive of outcomes related to our three themes (e.g., HDS Excitable and defensive; HDS Bold and denial; HDS Dutiful and superficial engagement) and items from these scales that directly aligned with each (e.g., “I can get angry quickly”, “I do many things better than almost everyone I know”, “I usually try to tell people what they want to hear”, respectively). Again, we wrote items designed to reflect what a person with a given tendency is likely to do when confronted with negative feedback about a failed decision. We piloted items and refined scales to improve psychometric qualities and avoid excessive overlap in content. We began writing items in the fall of 2011. We developed each scale by combining existing assessment content with new items, testing the scale with research samples, analyzing the data to calculate statistical properties, and making revisions to enhance the psychometric properties of each scale. We also received valuable input from colleagues across the globe. We tested this content using the same web platform previously described. Although we developed the pre- and post-decision scales using the same general process, they differ in two important ways. First, the pre-decision scales describe tendencies that are neither desirable nor undesirable, whereas low scores on the post-decision scales are undesirable. Second, we summarize the results differently. We present the results for pre-decision biases in terms of types, we present the results for post-decision reactions in terms of scale scores. 2.2 Definit ions of the Scales We define the scales for the three components of the Judgment Model as follows: Numerical Information Processing concerns the ability to quickly recognize patterns in numerical data and use those patterns effectively to make relevant decisions. Verbal Information Processing concerns the ability to quickly recognize and extract relevant facts from verbal information and use those facts effectively to make relevant decisions. Information Processing Styles describe a person’s preferences for acquiring and processing information needed to make decisions:

• Deliberate information processors take their time in processing both numerical and verbal information.

• Qualitative individuals are more efficient at processing verbal information than they are at processing numerical data.

• Quantitative individuals are more efficient at processing numerical data than they are at processing verbal information.

• Versatile information processors can quickly and effectively process both numerical data and verbal information.

Threat Avoidance vs. Reward Seeking concerns the degree to which an individual focuses on potential risks or possible rewards as a goal orientation when approaching decisions.


15

Tactical vs. Strategic Thinking concerns a person’s tendency to focus on immediate relevant details or larger, contextual issues. Data-Driven vs. Intuitive Decisions concerns the degree to which a person bases his or her decisions on relevant data or past experience. Decision-Making Styles are defined by combining scores across the three pre-decision scales. Doing this results in the following 8 decision-making types:

• Auditors avoid threats by making tactical decisions grounded in data. • Surgeons make tactical, intuitive decisions to avoid immediate threats. • Stock Traders seek rewards by making tactical, data-based decisions. • Defense Analysts avoid long-term threats by making strategic, data-based decisions. • Politicians seek long-term rewards by making strategic, intuitive decisions. • Chess Players defend against threats by making strategic, experience-based

decisions. • Promoters seek short-term rewards by making tactical, intuitive decisions. • Investors maximize long-term rewards by making strategic, data-based decisions.

Finally we come to reactions to feedback. There are three primary reactions as follows: Defensive vs. Cool-Headed concerns the degree to which individuals blame bad results on external factors or remain cool-headed and calmly reflect on their mistakes. Denial vs. Acceptance concerns the degree to which individuals respond to negative feedback by denying reality or deflecting problems versus directly addressing the failure. Superficial vs. Genuine Engagement concerns the degree to which individuals respond to negative feedback by putting on an act to gain approval versus actively engaging in feedback to improve future decisions. Scores on these three dimensions reflect: Openness to Feedback and Coaching, or an individual’s overall readiness to learn from mistakes and profit from coaching to improve future decision-making:

• Resistant individuals may tend to react to negative feedback about failed decisions by blaming others, denying responsibility, and pretending to care about feedback without actually engaging in it.

• Neutral individuals often seem moderately receptive to feedback, but may also struggle with tendencies to react poorly to news about failed decisions.

• Receptive individuals tend to analyze their missteps and solicit advice about how to make better decisions in response to a failed decision.


16

2.3 Item Composition of the Scales Each component of the Hogan Judgment Model is assessed using a distinct set of items. Below, we describe the item composition of the scales in each section. 2.3.1 Numerical Information Processing The numerical information processing assessment includes two parallel forms, each with 15 items of increasing difficulty. Participants have 10 minutes to answer as many questions as possible. Each item includes a stimulus, such as requiring the participant to complete sequences or identify patterns. Once the participant has examined the stimulus, he or she must make required computations and select the correct answer from a set of five possible responses. Participants can use scratch paper but not calculators or other external devices. Participants receive one point for each correct response and no points for incorrect responses, so total numerical scores range from 0 to 15. Scores of 0 to 7 indicate average numerical information processing ability; scores 8 to 15 indicate high ability. There is no item overlap and participants do not need previous knowledge or advanced education in mathematics to complete the assessment. 2.3.2 Verbal Information Processing The verbal information processing assessment includes two parallel forms, each containing 48 simple statements answered “True” or “False”. These items concern placement of letters in the alphabet, analogies, opposites, and item types, sizes, and meanings. Participants have two minutes to answer as many items as possible. They earn one point for each correct response, so total verbal scores range from 0 to 48. Scores of 0 to 28 indicate average verbal information processing ability; scores 29 to 48 indicate high ability. Readability statistics computed on the 48 items indicated an average sentence length of 7.2 words and an average word length of 3.5 characters. A Flesch-Kincaid reading level analysis shows that the verbal assessment is written at a first-grade level (1.4) with a reading ease of 97.5%, indicating that participants do not need previous knowledge or higher-order vocabulary to complete this assessment. 2.3.3 Decision-Making Approaches and Reactions to Feedback The decision-making style assessment presents items for both the pre-decision approach and the post-decision reaction scales. This 75 item assessment includes 28 items from the HPI, 19 from the HDS, 14 from the MVPI, and 14 new items. It includes simple statements participants answer as “True” if it describes them or “False” if it does not. We compute raw scores as the total number of items endorsed for the high end of each scale, and convert raw scores to normative scores based on a sample representative of the assessment’s intended population. For each scale, normative scores are presented in terms of a ratio indicating the participant’s natural underlying tendencies for both sides of the scale (e.g., 74% Strategic, 26% Tactical). There is no item overlap across the 75 items, which were screened for content participants might perceive as offensive or invasive. Specifically, there are no items concerning sexual


17

preferences, religious or political beliefs, criminal or otherwise illegal behavior, racial/ethnic attitudes, or perceptions of disabled individuals. Readability statistics computed on the 75 items indicated an average sentence length of 8.3 words and an average word length of 4.1 characters. A Flesch-Kincaid reading level analysis shows that the decision-making style assessment is written at a fourth-grade level (4.3) with a reading ease of 78.4%. 2.4 Descriptive Statistics and Reliabil it ies Table 2.1 presents descriptive statistics for each of the scales in the Hogan Judgment Model, including the number of items in each scale, mean scale scores, standard deviations, skewness and kurtosis statistics, internal consistency reliability, average inter-item correlations, and standard errors of measurement. Skewness refers to departure from symmetry in a distribution of scores. When a distribution is normal and symmetrical, skewness values are around zero. Positive skewness values indicate that most scores fall at the bottom end of a distribution, and negative skewness values indicate that most scores fall on the top end of a distribution. Skewness values greater than +1.0 or less than -1.0 generally indicate a significant departure from symmetry. Kurtosis refers to how peaked or flat a score distribution is relative to the normal distribution. When scores are normally distributed, kurtosis values are around zero and the score distribution is said to be “mesokurtic”. When the distribution is sharper than the normal distribution, kurtosis values are positive and the score distribution is said to be “leptokurtic”. When the distribution is broader than the normal distribution, kurtosis values are negative and the score distribution is said to be “platykurtic”. Kurtosis values of more than twice the standard error indicate a significant departure from the normal distribution. Internal consistency is a form of reliability estimating how well items from a single scale estimate a common attribute. When items measure the same construct, internal consistency reliability is high. When items measure different constructs, it is low. Average correlations between items also provide information about internal consistency (DeVellis, 1991), where higher average inter-item correlations indicate that items measure the same construct. Clark and Watson (1995) suggest that average inter-item correlations be at least .15, although acceptable thresholds may be lower depending on the heterogeneity of the construct. We also provide two estimates for standard error of measurement – one based on observed scores (SEM1), and the other (SEM3) based on true score estimates (Dudek, 1979). Standard error of measurement represents the standard deviation of a person’s scores if he or she were to complete the assessment repeatedly. Lower estimates indicate the results are more stable. To examine the descriptive statistics and reliability, we obtained data from a global sample of executives, managers, and other high-level professionals who completed the instrument as part of leadership development or research. On average, participants were 54.96 years


18

old (SD = 17.03), and the sample included 112 males and 171 females (248 did not indicate their sex). Table 2.1 presents descriptive statistics, reliability estimates, and standard errors. Table 2.1 Classical Scale Statistics for Hogan Judgment Model Scales Scale Items M SD Skew Kurt α Avg. r SEM1 SEM3 Numerical 15 7.32 2.50 .02 .06 .63 .10 1.52 1.94 Verbal ^ 48 20.84 6.20 .28 .00 .94 .26 1.52 2.12 Threat vs. Reward 14 8.01 3.46 -.34 -.63 .80 .22 1.55 2.08 Tactical vs. Strategic 13 8.21 3.42 -.51 -.63 .82 .26 1.45 1.96 Data-Driven vs. Intuitive 12 5.61 3.40 .16 -1.06 .82 .28 1.44 1.95 Defensive vs. Cool-Headed 13 8.82 3.21 -.85 .06 .80 .25 1.44 1.93 Denial vs. Acceptance 11 4.93 2.65 .23 -.66 .73 .20 1.38 1.81 Superficial vs. Genuine 12 7.28 2.69 -.40 -.34 .69 .16 1.50 1.95 Note: N = 502; ^ = Verbal item sub-sample N = 135; Items = Number of items in each scale; M = Scale mean; SD = Standard deviation; Skew = Skewness statistic; Kurt = Kurtosis statistic; α = Cronbach’s alpha internal consistency reliability; Avg. r = Average inter-item correlation; SEM1 = Standard error of measurement applied to an individual’s estimated true score given their observed score; SEM3 = Standard error of measurement applied to an individual’s observed score. For numerical and verbal information processing, average scores are near scale midpoints, show adequate variability, and demonstrate no significant departures from symmetry. Reliability estimates and average inter-item correlations indicate both scales are internally consistent and conceptually related. For the pre-decision scales, mean scores range from 5.61 (Data-Driven vs. Intuitive Decisions) to 8.21 (Tactical vs. Strategic Thinking), with standard deviations between 3.40 (Data-Driven vs. Intuitive Decisions) and 3.46 (Threat Avoidant vs. Reward Seeking). Skewness statistics indicate that score distributions are adequately symmetrical, ranging from -.51 (Tactical vs. Strategic Thinking) to .16 (Data-Driven vs. Intuitive Decisions). Kurtosis statistics also indicate that the pre-decision scales are not abnormally peaked or flat, ranging from -1.06 (Data-Driven vs. Intuitive Decisions) to -.63 (Threat Avoidant vs. Reward Seeking; Tactical vs. Strategic Thinking). Cronbach’s alpha reliability estimates illustrate that these scales are internally consistent, ranging from .80 (Threat Avoidant vs. Reward Seeking) to .82 (Tactical vs. Strategic Thinking; Data-Driven vs. Intuitive Decisions), with an average reliability of .81 across scales. Average inter-item correlations range from .22 (Threat Avoidant vs. Reward Seeking) to .28 (Data-Driven vs. Intuitive Decisions), averaging .25 across scales. As a whole, these results provide support for the psychometric qualities of the pre-decision scales. For the post-decision scales, mean scores range from 4.93 (Denial vs. Acceptance) to 8.82 (Defensive vs. Cool-Headed), with standard deviations between 2.65 (Denial vs. Acceptance) and 3.21 (Defensive vs. Cool-Headed). Skewness statistics indicate that score distributions are adequately symmetrical, ranging from -.85 (Defensive vs. Cool-Headed) to .23 (Denial vs. Acceptance). Kurtosis statistics also indicate that the post-decision scales are not abnormally peaked or flat, ranging from -.66 (Denial vs. Acceptance) to .06 (Defensive vs. Cool-Headed). Cronbach’s alpha reliability estimates illustrate that these scales are internally consistent, ranging from .69 (Superficial vs. Genuine Engagement) to .80


19

(Defensive vs. Cool-Headed), with an average reliability of .74 across scales. Average inter-item correlations range from .16 (Superficial vs. Genuine Engagement) to .25 (Defensive vs. Cool-Headed), averaging .20 across scales. These results provide support for the psychometric qualities of the post-decision scales. The following section presents evidence for the parallel forms used to test numerical and verbal information processing. 2.5 Parallel Forms Evidence for Numerical and Verbal Scales As previously noted, we developed two parallel forms to assess numerical and verbal information processing. When using parallel forms, it is necessary to demonstrate that they are parallel so that participants are likely to receive the same score regardless of which form they complete. To ensure that the forms of our numerical and verbal assessments would be parallel, we first developed parallel items. For the verbal section, both forms rely on identical items that are presented in a different order across forms within blocks of five. This ensures that participants view the same items, and helps maintain item security because participants cannot simply memorize one set of correct responses. We will continue to create additional parallel forms as needed using this same process in the future. For the numerical section, we created one set of items and slightly modified the numbers and/or patterns in each item so that arriving at the correct answer across forms required the same logic and steps. We created parallel distractor responses using this same approach, but presented distractors in a different order to ensure that participants could not simply memorize one set of correct responses. To examine the effectiveness of these efforts, we compared descriptive statistics and internal consistency estimates of reliability for each set of parallel forms, and conducted independent samples t-tests on mean scores from our research sample. Tables 2.2 and 2.3 present these results. Table 2.2 Parallel Forms Evidence for Numerical Information Processing

Numerical Form N Min Max M SD Skew Kurt α t-test t df ρ

Form 1 237 2 15 7.44 2.42 .18 -.23 .64 1.01 501 .31 Form 2 266 0 15 7.22 2.56 -.10 .22 .63 Note: N = Sample size; Min = Minimum score; Max = Maximum score; M = Mean; SD = Standard deviation; Skew = Skewness statistic; Kurt = Kurtosis statistic; α = Cronbach’s alpha internal consistency reliability; t = t-statistic from independent samples t-test; df = degrees of freedom; ρ = significance level for t-statistic.


20

Table 2.3 Parallel Forms Evidence for Verbal Information Processing

Verbal Form N Min Max M SD Skew Kurt α t-test t df ρ

Form 1 64 8 35 20.58 6.34 .05 -.42 .92 -.47 133 .64 Form 2 71 8 37 21.08 6.10 .54 .46 .92 Note: N = Sample size; Min = Minimum score; Max = Maximum score; M = Mean; SD = Standard deviation; Skew = Skewness statistic; Kurt = Kurtosis statistic; α = Cronbach’s alpha internal consistency reliability; t = t-statistic from independent samples t-test; df = degrees of freedom; ρ = significance level for t-statistic. These results reveal no significant differences in scale means, variance, or distributions, indicating that the small differences that exist across forms for both assessments are simply the result of sampling error. Therefore, no single form of the numerical and verbal assessments offers participants an advantage, and the use of parallel forms helps maintain assessment security. 2.6 Intercorrelations Between Hogan Judgment Model Scales We computed correlations between the scales using the global sample of executives, managers, and other high-level professionals previously described. Table 2.4 presents these correlations. Table 2.4 Correlations between Hogan Judgment Model Scales Scale 1 2 3 4 5 6 7 8 1. Numerical Info. Processing 1.00 .45** .17** .10* -.16** .07 -.10* .10* 2. Verbal Info. Processing 1.00 .10* .06 -.10* .01 -.07 .08 3. Threat vs. Reward 1.00 .45** -.19** .13** -.36** .34** 4. Tactical vs. Strategic 1.00 -.08 .11* -.67** .47** 5. Data-Driven vs. Intuitive 1.00 -.09* .11* -.12** 6. Defensive vs. Cool-Headed 1.00 .00 .16** 7. Denial vs. Acceptance 1.00 -.32** 8. Superficial vs. Genuine 1.00 Note: N = 502; *Correlation is significant at .05 level; **Correlation is significant at .01 level. Results reveal five points worth discussion. First, the numerical and verbal information processing scales correlate at .45, indicating they are conceptually-related, yet independent, components of cognitive ability. Second, the magnitudes of correlations between the cognitive and non-cognitive scales of the model range from -.16 to .17, suggesting little connection between these components. Third, correlations between pre-decision scales (scales 3-5) range from -.19 to .45. Among these scales, the relationship between the Threat Avoidance vs. Reward Seeking and Tactical vs. Strategic Thinking scales stands out. However, this association makes intuitive sense, because organizational leaders often focus on long-term rewards to succeed. Fourth, correlations between post-decision scales (scales 6-8) range from -.32 to .16. Again, this pattern of relationships suggests that these dimensions are moderately related, yet measure independent constructs. Finally, correlations between the pre-decision and post-


21

decision scales range between -.67 and .47, suggesting (as one would expect) that people’s decision-making tendencies are somewhat related to how they react to feedback about failed decisions. Specifically, the -.67 correlation between the Tactical vs. Strategic Thinking scale and the Denial vs. Acceptance scale suggests that strategic decision-makers tend to make big picture decisions and move on, whereas tactical decision-makers may be more interested in revisiting their decisions and diving into the details of feedback. Although the scales in our model are moderately correlated results in Table 2.4 suggest that the scales represent unique and independent predictors of business judgment. 2.7 Structural Psychometrics of the Pre- and Post-Decision Scales Factor analysis is a statistical methodology designed to account for the relationships between many variables using a fewer number of factors. A factor represents something shared in common by different variables; it is a linear combination of items which together measure a single construct. Thus, this technique allows us to examine whether responses to different items cluster together into meaningful factors and make sense of the many relationships between individual assessment items. We conducted an exploratory Principal Components Analysis (PCA) to identify underlying factors of the pre- and post-decision scales. We used a sample of 872 employed adults. Table 2.5 provides age, gender, and racial/ethnic data for this sample. Because the pre- and post-decision scales target two different contexts, we conducted separate PCAs for pre- and post-decision scales. We chose the number of components to extract based on the size of the Eigenvalues and an examination of several alternate solutions. Finally, we refined the components using orthogonal Varimax rotation to find the most economical solution with the goal of associating each item to one factor. Table 2.5 Sample Ethnic Composition by Age and Gender Age Under 40 40 and Over Gender Male Female Male Female Race/Ethnicity N % N % N % N % Two or More Races 3 0.34 7 0.80 0 0.00 0 0.00 Black/African-American 12 1.38 18 2.06 1 0.11 4 0.46 Hispanic/Latino 14 1.61 13 1.49 2 0.23 2 0.23 Asian 28 3.21 14 1.61 3 0.34 0 0.00 American Indian/Alaska Native 2 0.23 4 0.46 4 0.46 1 0.11 White 198 22.71 178 20.41 57 6.54 56 6.42 Not Indicated 2 0.23 10 1.15 0 0.00 1 0.11 Native Hawaiian/Other Pacific Islander 2 0.23 0 0.00 0 0.00 0 0.00 TOTAL 261 29.93 244 27.98 67 7.68 64 7.34 Note: 37 participants Under 40 and 1 participant 40 and Over did not identify their gender.


22

2.7.1 Pre-Decision Scales Table 2.6 presents PCA results for the pre-decision scales. To protect the proprietary nature of our intellectual property, the table does not include text for specific assessment items. Table 2.6 Varimax Rotated Factor Matrix for Pre-Decision Scales Hogan Judgment Assessment Pre-Decision Scale Item Factor I Factor II Factor III Threat Avoidance vs. Reward Seeking Item 1 -0.59 Threat Avoidance vs. Reward Seeking Item 2 -0.63 Threat Avoidance vs. Reward Seeking Item 3 -0.67 Threat Avoidance vs. Reward Seeking Item 4 -0.57 Threat Avoidance vs. Reward Seeking Item 5 -0.56 Threat Avoidance vs. Reward Seeking Item 6 -0.37 Threat Avoidance vs. Reward Seeking Item 7 -0.46 Threat Avoidance vs. Reward Seeking Item 8 0.37 0.47 Threat Avoidance vs. Reward Seeking Item 9 0.33 Threat Avoidance vs. Reward Seeking Item 10 0.44 Threat Avoidance vs. Reward Seeking Item 11 0.55 Threat Avoidance vs. Reward Seeking Item 12 0.47 Threat Avoidance vs. Reward Seeking Item 13 0.64 Threat Avoidance vs. Reward Seeking Item 14 0.66 Tactical vs. Strategic Thinking Item 1 0.77 Tactical vs. Strategic Thinking Item 2 0.59 Tactical vs. Strategic Thinking Item 3 0.67 Tactical vs. Strategic Thinking Item 4 -0.65 Tactical vs. Strategic Thinking Item 5 0.66 Tactical vs. Strategic Thinking Item 6 0.73 Tactical vs. Strategic Thinking Item 7 0.63 Tactical vs. Strategic Thinking Item 8 -0.55 Tactical vs. Strategic Thinking Item 9 -0.55 Tactical vs. Strategic Thinking Item 10 -0.64 Tactical vs. Strategic Thinking Item 11 0.44 0.31 Tactical vs. Strategic Thinking Item 12 0.77 Tactical vs. Strategic Thinking Item 13 -0.68 Data-Driven vs. Intuitive Item 1 -0.40 Data-Driven vs. Intuitive Item 2 -0.31 Data-Driven vs. Intuitive Item 3 0.69 Data-Driven vs. Intuitive Item 4 0.71 Data-Driven vs. Intuitive Item 5 0.76 Data-Driven vs. Intuitive Item 6 0.56 Data-Driven vs. Intuitive Item 7 0.73 Data-Driven vs. Intuitive Item 8 0.78 Data-Driven vs. Intuitive Item 9 0.68 Data-Driven vs. Intuitive Item 10 0.73 Data-Driven vs. Intuitive Item 11 0.59 Data-Driven vs. Intuitive Item 12 0.43 Results support the expected three-factor solution. Items for each scale loaded onto a single factor (e.g., all Tactical/Strategic items loaded onto one factor). Only two items demonstrated cross-loadings onto a second factor, but in both cases, the highest loading for the item was still on the intended factor.


23

2.7.2 Post-Decision Scales Table 2.7 presents PCA results for the post-decision scales. Again, to protect the proprietary nature of our intellectual property, the table does not include text for specific assessment items. Table 2.7 Varimax Rotated Factor Matrix for Post-Decision Scales Hogan Judgment Assessment Post-Decision Scale Item Factor I Factor II Factor III Defensive vs. Cool-Headed Item 1 0.56 Defensive vs. Cool-Headed Item 2 0.60 Defensive vs. Cool-Headed Item 3 0.64 Defensive vs. Cool-Headed Item 4 -0.73 Defensive vs. Cool-Headed Item 5 0.62 Defensive vs. Cool-Headed Item 6 -0.73 Defensive vs. Cool-Headed Item 7 0.53 Defensive vs. Cool-Headed Item 8 0.68 Defensive vs. Cool-Headed Item 9 0.76 Defensive vs. Cool-Headed Item 10 0.71 Defensive vs. Cool-Headed Item 11 0.43 Defensive vs. Cool-Headed Item 12 -0.65 Defensive vs. Cool-Headed Item 13 -0.55 Denial vs. Acceptance Item 1 0.57 Denial vs. Acceptance Item 2 0.56 Denial vs. Acceptance Item 3 0.66 Denial vs. Acceptance Item 4 0.62 Denial vs. Acceptance Item 5 0.51 Denial vs. Acceptance Item 6 0.59 Denial vs. Acceptance Item 7 0.54 Denial vs. Acceptance Item 8 0.68 Denial vs. Acceptance Item 9 0.63 Denial vs. Acceptance Item 10 0.53 Denial vs. Acceptance Item 11 0.55 Superficial vs. Genuine Engagement Item 1 0.56 Superficial vs. Genuine Engagement Item 2 0.59 Superficial vs. Genuine Engagement Item 3 0.52 Superficial vs. Genuine Engagement Item 4 0.63 Superficial vs. Genuine Engagement Item 5 0.52 Superficial vs. Genuine Engagement Item 6 0.63 Superficial vs. Genuine Engagement Item 7 0.44 Superficial vs. Genuine Engagement Item 8 0.44 Superficial vs. Genuine Engagement Item 9 0.39 -0.30 Superficial vs. Genuine Engagement Item 10 0.42 Superficial vs. Genuine Engagement Item 11 0.53 Superficial vs. Genuine Engagement Item 12 0.39 -0.35 Again, results generally support a three-factor solution. All but one item (#7 from the Superficial vs. Genuine Engagement scale) loaded most strongly onto its intended factor, and as with the pre-decision scales, there were only two items with cross-loadings. Although adequate, these results demonstrate the need to continue examining the factor structure of the scales as additional data become available and revising selected items as needed.

3. VALIDITY

24

3. Validity 3.1 Construct Validity Validity concerns the meaning of scale scores. We believe the purpose of assessment is to predict significant non-test outcomes. Following Gough (1975), we designed the Hogan Judgment Report to predict how people process information, make decisions, and react to feedback and coaching in work settings. Therefore, the validity of the Hogan Judgment Report scales depends on empirical data that make sense given our theory of each scale’s content (also see R. Hogan, Hogan, & Roberts, 1996). Such information includes correlations with assessments measuring personality, values/ needs/motives/interests, and cognitive ability. The following section presents selected results from 16 such assessments. Appendix B presents full correlation matrices between the Hogan Judgment Report scales and these assessments. 3.1.1 Procedure We obtained these data through Goldberg’s (2008) longitudinal research with the Eugene-Springfield Community Sample (ESCS). Goldberg recruited approximately 1,000 people to participate in the project. However, only a portion of this sample completed both the judgment assessment and the others we used to examine construct validity. We also collected data using online, unproctored Internet testing. Subjects received compensation for their participation. No participants took the assessment as part of high stakes testing where results impacted hiring, promotion, or other personnel decisions. In the following section, we report the sample composition for each assessment. 3.1.2 Samples and Instruments HPI. First, we review judgment scale correlations with scales and subscales from the Hogan Personality Inventory (HPI; R. Hogan & Hogan, 1995, 2007; see Table B.1). Ninety-eight participants completed the Hogan Judgment Report and the HPI. The sample included 43 males and 55 females. Ages of subjects ranged from 29 years to 72 years with a mean of 46.31 years (SD = 8.00). The HPI is a 206-item True/False measure of normal personality, which evolved from the Five-Factor Model (FFM; De Raad & Perugini, 2002; Wiggins, 1996) and socioanalytic theory (R. Hogan, 1983, 1991, 1996). Hogan and Holland (2003, p. 104) demonstrate that the HPI is an adequate measure of the FFM, with median correlations with other FFM inventories ranging from .30 to .69. The global HPI normative group includes 144,877 working adults representing the global workforce (R. Hogan & Hogan, 2007). The HPI contains seven primary scales and a validity scale. In addition, a number of occupational scales are available for specialized applications. The seven primary scales are Adjustment (ADJ), Ambition (AMB), Sociability

3. VALIDITY

25

(SOC), Interpersonal Sensitivity (INP), Prudence (PRU), Inquisitive (INQ), and Learning Approach (LRN). The validity key (VAL) contains 14 items and detects careless or random responding. The HPI technical manual (R. Hogan & Hogan, 2007) presents reliability, confirmatory factor analysis, and validity. The Mental Measurements Yearbook (Lobello, 1996) and the British Psychological Society’s Psychological Testing Centre’s test reviews (British Psychological Society, 2007) also provide professional reviews of the HPI. HDS. Second, we review judgment scale correlations with scales and subscales of the Hogan Development Survey (HDS; R. Hogan & Hogan, 2009; Hogan Assessment Systems, 2014; see Table B.2). Eighty-five participants completed the Hogan Judgment Report and the HDS. The sample included 41 males and 44 females. Ages of subjects ranged from 21 years to 67 years with a mean of 44.35 years (SD = 8.85). The HDS concerns characteristics that can derail careers, relationships, and other productive activities. The global HDS normative sample includes 67,614 working adults representing the global workforce (R. Hogan & Hogan, 2009). The HDS contains 11 primary scales: Excitable (EXC), Skeptical (SKE), Cautious (CAU), Reserved (RES), Leisurely (LEI), Bold (BOL), Mischievous (MIS), Colorful (COL), Imaginative (IMA), Diligent (DIL), and Dutiful (DUT). These scales assess dysfunctional dispositions that emerge when people stop considering how their actions affect others. Over time, these dispositions create a person’s reputation and can impede job performance and career success. The HDS is not a medical or clinical assessment. It does not measure personality disorders, which are manifestations of mental disorder. Instead, the HDS assesses self-defeating expressions of normal personality. The DSM-5 (American Psychiatric Association, 2013 p.647) makes this same distinction between behavioral traits and disorders – self-defeating behaviors, such as those predicted by the HDS, come and go depending on the context. In contrast, personality disorders are enduring and pervasive across contexts. The HDS technical manual (R. Hogan & Hogan, 2009) and the HDS Form 5 technical supplement (Hogan Assessment Systems, 2014) provide more details about the reliability, validity, factor structure, and norm development of the assessment. CPI. Third, we review judgment scale correlations with scales from the California Psychological Inventory (CPI; Gough, 1996; see Table B.3). One hundred and ninety-five participants completed the Hogan Judgment Report and the CPI. The sample included 88 males and 107 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.71 years (SD = 8.78). The CPI is a 434-item True/False inventory of personality. The CPI is normed on 52 male and 42 female samples, including high school, college, graduate, professional school, and occupational samples. The CPI contains 20 folk and 3 vector scales. The 20 folk scales are Dominance (Do), Capacity for Status (Cs), Sociability (Sy), Social Presence (Sp), Self-acceptance (Sa), Independence (In), Empathy (Em), Responsibility (Re), Socialization (So), Self-control (Sc), Good Impression (GI), Communality (Cm), Well-being (Wb), Tolerance (To), Achievement via Conformance (Ac), Achievement via Independence (Ai), Intellectual Efficiency (Ie), Psychological-mindedness (Py), Flexibility (Fx), and Femininity/Masculinity

3. VALIDITY

26

(F/M). The three vector scales are Externality/Internality (v.1), Norm-doubting/Norm-favoring (v.2), and Ego-integration (v.3). The CPI technical manual (Gough, 1996) provides details about reliability, factor analysis, and validity of the assessment. NEO PI-R. Fourth, we review judgment scale correlations with scales and facets from the NEO PI-R (Costa & McCrae, 1992; see Table B.4). One hundred and eighty-two participants completed the Hogan Judgment Report and the NEO PI-R. The sample included 82 males and 100 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.96 years (SD = 8.89). The NEO PI-R is a 240-item True/False measure of personality. Specifically, it measures five major dimensions of personality, as well as important facets of each domain with applicability in both clinical and research domains. The NEO PI-R norming sample includes 1,000 adults, representing a stratification of the U.S. population based on race and age (Costa & McCrae, 1992). The NEO PI-R contains 5 domain scales and 30 facet scales. The 5 domains are Neuroticism (N), Extraversion (E), Openness (O), Agreeableness (A), and Conscientiousness (C). Six facet scales fall within each domain. The Neuroticism facets are Anxiety (N1), Angry Hostility (N2), Depression (N3), Self-Consciousness (N4), Impulsiveness (N5), and Vulnerability (N6). The Extraversion facets are Warmth (E1), Gregariousness (E2), Assertiveness (E3), Activity (E4), Excitement-Seeking (E5), and Positive Emotions (E6). The Openness facets are Fantasy (O1), Aesthetics (O2), Feelings (O3), Actions (O4), Ideas (O5), and Values (O6). The Agreeableness facets are Trust (A1), Straightforwardness (A2), Altruism (A3), Compliance (A4), Modesty (A5), and Tender-Mindedness (A6). The Conscientiousness facets are Competence (C1), Order (C2), Dutifulness (C3), Achievement Striving (C4), Self-Discipline (C5), and Deliberation (C6). The NEO PI-R technical manual (Costa & McCrae, 1992) provides details about the reliability, factor analysis, and validity of the assessment. IPIP. Fifth, we review judgment scale correlations with the International Personality Item Pool (IPIP; Goldberg, Johnson, Eber, Hogan, Ashton, Cloninger, & Gough, 2006; see Table B.5). One hundred and sixty-nine participants completed the Hogan Judgment Report and the IPIP. The sample included 76 males and 93 females. Ages of subjects ranged from 22 years to 72 years with a mean of 45.86 years (SD = 8.71). The International Personality Item Pool (IPIP; Goldberg, 1999; Goldberg, et al., 2006) is an online pool of over 2,000 personality assessment items. The purpose of IPIP is to continuously develop and refine personality inventories. The IPIP is available for anyone to contribute and/or use. Currently, users can create 269 scales using the available items. The IPIP and corresponding scales are updated regularly to use new and refined items. We presented correlations using the following scales: Extraversion (EXT), Agreeableness (AGR), Conscientiousness (CON), Emotional Stability (EMS), and Intellect/Imagination (I/I). Goldberg et al. (2006) provides technical features of the IPIP, including norm samples, scale construction, and validity indices. The IPIP website (http://ipip.ori.org) provides additional information. 16PF. Sixth, we review judgment scale correlations with scales from the Sixteen Personality Factor Questionnaire (16PF; Conn & Rieke, 1994; Russell & Karol, 2002; see Table B.6).

3. VALIDITY

27

One hundred and ninety-seven participants completed the Hogan Judgment Report and the 16PF. The sample included 89 males and 108 females. Ages of subjects ranged from 22 years to 72 years with a mean of 45.55 years (SD = 8.39). The 16PF is a 185-item measure of normal personality, whose foundation is based on factor analyzing all English-language adjectives describing human behavior. The 16PF norming sample includes 10,261 adults representing a stratification of the U.S. adult population. The assessment contains 16 bipolar scales with meaning for both high and low scale scores. In addition, there are five global factor scales and an impression management index assessing social desirability. The 16 factor scales are Factor A: Warmth, Factor B: Reasoning, Factor C: Emotional Stability, Factor E: Dominance, Factor F: Liveliness, Factor G: Rule-Consciousness, Factor H: Social Boldness, Factor I: Sensitivity, Factor L: Vigilance, Factor M: Abstraction, Factor N: Privateness, Factor O: Apprehension, Factor Q1: Openness to Change, Factor Q2: Self-Reliance, Factor Q3: Perfectionism, and Factor Q4: Tension. The five global factor scales are Extraversion (EX), Anxiety (AX), Tough-mindedness (TM), Independence (IN), and Self-Control (SC). Conn and Rieke (1994) provide information on the technical features of the assessment, including reliability, item analysis, factor analysis, and validity. TCI. Seventh, we review judgment scale correlations with scales from the Temperament and Character Inventory (TCI; see Table B.7). Two hundred and three participants completed the Hogan Judgment Report and the TCI. The sample included 91 males and 112 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.45 years (SD = 8.71). The TCI is a 295-item adaptation of Cloninger, Przybeck, Svrakic, and Wetzel's (1994) multi-level personality inventory, with 35 lower-level scales subsumed by 7 higher-level constructs. The four temperament dimensions are: Novelty Seeking (NS; Exploratory excitability vs. Stoic rigidity, Impulsiveness vs. Reflection, Extravagance vs. Reserve, Disorderliness vs. Regimentation); Harm Avoidance (HA; Anticipatory worry and pessimism vs. Uninhibited optimism, Fear of uncertainty, Shyness with strangers, Fatigability and asthenia); Reward Dependence (RD; Sentimentality, Openness to warm communication vs. Aloofness, Attachment, Dependence); and Persistence (PS; Eagerness of effort vs. Laziness, Work-hardened vs. Spoiled, Ambitious vs. Underachieving, Perfectionist vs. Pragmatist). The three character dimensions are: Self-Directedness (SD; Responsibility vs. Blaming, Purposefulness vs. Lack of goal-orientation, Resourcefulness, Self-acceptance vs. Self-striving, Enlightened Second Nature); Cooperativeness (CO; Social acceptance vs. Social intolerance, Empathy vs. Social disinterest, Helpfulness vs. Unhelpfulness, Compassion vs. Revengefulness, Pure-hearted Conscience vs. Self-serving advantage); and Self-Transcendence (ST; Self-forgetful vs. Self-conscious experience, Transpersonal identification vs. Self-differentiation, Spiritual acceptance vs. Rational materialism; Enlightened vs. Objective, Idealistic vs. Practical). The TCI website (https://tci.anthropedia.org/en/) provides information on the technical features of the assessment. MPQ. Eighth, we review judgment scale correlations with scales from the Multidimensional Personality Questionnaire (MPQ; Tellegen & Waller, 2008; see Table B.8). Two hundred and five participants completed the Hogan Judgment Report and the MPQ. The sample included

3. VALIDITY

28

93 males and 112 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.58 years (SD = 8.65). The MPQ contains 11 primary trait dimension scales. Most of the 276 MPQ items use a True/False response format. The norming sample comes from the Minnesota Twins Registry, which is a community sample of 1,350 individuals between 20 and 60 years of age. The sample includes 675 males and 675 females. The 11 primary trait scales of the MPQ include Wellbeing (WB), Social Potency (SP), Achievement (AC), Social Closeness (SC), Stress Reaction (SR), Aggression (AG), Alienation (AL), Control (CO), Harm Avoidance (HA), Traditionalism (TR), and Absorption (AB). Tellegen and Waller (2008) provide information on the technical features of the assessment, including item analysis, reliability, factor analysis, and validity. 6FPQ. Ninth, we review judgment scale correlations with factor and facet scores from the Six Factor Personality Questionnaire (6FPQ; Jackson, Paunonen, Fraboni, & Goffin, 1996; see Table B.9). Two hundred participants completed the Hogan Judgment Report and the 6FPQ. The sample included 90 males and 110 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.44 years (SD = 8.53). The 6FPQ extends FFM personality dimensions to measure six dimensions: Extraversion, Agreeableness, Independence, Openness to Experience, Methodicalness, and Industriousness. Specifically, the 6FPQ divides Conscientiousness into Methodicalness and Industriousness, and redefines Neuroticism as the opposite pole of the dimension, Independence. The norm group includes a random sample of 483 males and 584 females in the U.S. and Canada. The test publisher also provides a norm sample of 4,040 job applicants to support the workforce application of the assessment. The 6FPQ includes six factor scales, each with three subscales: Extraversion (Affiliation; Dominance; Exhibition); Agreeableness (Abasement; Even-tempered; Good-natured); Independence (Autonomy; Individualism; Self-reliance); Openness to Experience (Change; Understanding; Breadth of Interest); Methodicalness (Cognitive Structure; Deliberateness; Order); and Industriousness (Achievement; Endurance; Seriousness). The 6FPQ technical manual (Jackson, 1984) and relevant research studies (e.g., Jackson, Paunonen, Fraboni, & Goffin, 1996) provide information on the technical features of the assessment, including reliability, factor analysis, and validity. PRS. Tenth, we review judgment scale correlations with personality content from the Personal Reactions Survey (PRS; see Table B.10). Two hundred and eight participants completed the Hogan Judgment Report and the PRS. The sample included 94 males and 114 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.46 years (SD = 8.68). The PRS includes 192 items from the HEXACO Personality Inventory (Lee & Ashton, 2004), 20 items from four Behavioral Activation System (BAS) and Behavioral Inhibition System (BIS) scales from Carver and White (1994), and 23 items selected by Jackson (2003) as markers of the major factors in the Gray-Wilson Personality Questionnaire (Wilson, Barrett, &

3. VALIDITY

29

Gray, 1989). However, to validate the Hogan Judgment Report scales, our analyses focused on the BAS, BIS, and HEXACO scales. MVPI. Eleventh, we review judgment scale correlations with scales and themes from the Motives, Values, Preferences Inventory (MVPI; J. Hogan & Hogan, 2010; see Table B.11). Ninety-nine participants completed the Hogan Judgment Report and the MVPI. The sample included 44 males and 55 females. Ages of subjects ranged from 30 years to 68 years with a mean of 45.31 years (SD = 8.16). The MVPI is a 200-item measure of values and preferences which assesses individual motives and evaluates person-organization fit. The global MVPI normative sample includes 48,267 working adults representing the global workforce (J. Hogan & Hogan, 2010). The MVPI contains 10 primary scales: Aesthetics (AES), Affiliation (AFF), Altruism (ALT), Commerce (COM), Hedonism (HED), Power (POW), Recognition (REC), Science (SCI), Security (SEC), and Tradition (TRA). Each scale also reflects five themes: Lifestyles, Beliefs, Occupational Preferences, Aversions, and Preferred Associations. The MVPI technical manual (J. Hogan & Hogan, 2010) provides details about item content, reliability, factor analysis, and validity of the instrument. CISS. Twelfth, we review judgment scale correlations with interest and skill scales from the Campbell Interest and Skill Survey (CISS; Campbell, Hyne, & Nilsen, 1992; see Table B.12). One hundred and sixty-six participants completed the Hogan Judgment Report and the CISS. The sample included 76 males and 90 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.03 years (SD = 8.80). The CISS maps self-reported skills and interests to the occupational world to provide individuals with career guidance. It is intended for use with most adults and students as young as 15. This survey contains 200 interest items and 120 skill items measured on a six-point scale. The interest scales provide an indicator for the strength of attraction to occupational areas, while the skills scales are an estimate of competence. Overall, there are seven major orientation scales (i.e., Influencing, Organizing, Helping, Creating, Analyzing, Producing, and Adventuring), which indicate attraction to and confidence in each orientation. Individuals receive both interest and skill scores for each orientation, and can score either high or low on both, providing four possible interest and skill combinations for each orientation. There are also 29 basic interest and skill subscales for the orientations, covering specific topics such as public speaking or mathematics. The CISS norming sample includes 5,000 people from over 60 different occupations. Campbell, Hyne, and Nilsen (1992) provide information on the technical features of the assessment, including item analysis, scale construction, reliability, and validity. JPI-R. Thirteenth, we review judgment scale correlations with scales from the Jackson Personality Inventory - Revised (JPI-R; Jackson, 1994; see Table B.13). Two hundred and seven participants completed the Hogan Judgment Report and the JPI-R. The sample included 94 males and 113 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.45 years (SD = 8.69).

3. VALIDITY

30

The JPI-R is a 300-item True/False measure of personality concerning individuals’ interpersonal patterns of interaction, cognitive styles, and value orientations. The JPI-R is normed on four different populations: high school students, blue-collar workers, executives, and adult college students. The college, blue collar, and executive norm groups are combined to form an overall norm group of 1,436 individuals (Jackson, 1994). The JPI-R contains 15 content scales: Complexity (Cpx), Breadth of Interest (Bdi), Innovation (Inv), Tolerance (Tol), Empathy (Emp), Anxiety (Axy), Cooperativeness (Cpr), Sociability (Soc), Social Confidence (Scf), Energy Level (Enl), Social Astuteness (Sas), Risk Taking (Rkt), Organization (Org), Traditional Values (Trv), and Responsibility (Rsy). These scales are grouped into five meaningful clusters: Analytical, Emotional, Extroverted, Opportunistic, and Dependable. Jackson (1994) provides information on the technical features of the assessment, including item analysis, reliability, factor analysis, and validity. HBRI. Fourteenth, we review judgment scale correlations with scales from the Hogan Business Reasoning Inventory (HBRI; R. Hogan, Barrett, & Hogan, 2007; see Table B.14). Two hundred and five participants completed the Hogan Judgment Report and the HBRI. The sample included 94 males and 111 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.44 years (SD = 8.74). The HBRI is a 24-item measure of cognitive skills and business reasoning, intended for use with college-educated (i.e., bachelor’s degree and above) managers and professionals. Items reflect cognitive tasks with content similar to that used in actual business operations. The HBRI norming sample includes 2,484 adults who completed the assessment as part of applicant screening, employee development, or research purposes (R. Hogan, Barrett, & Hogan, 2007). The HBRI includes strategic reasoning and tactical reasoning scales, which are combined to examine overall critical reasoning. The HBRI technical manual (R. Hogan, Barrett, & Hogan, 2007) provides information on the technical features of the test, including reliability, scale construction, factor analysis, and validity. SPA. Fifteenth, we review judgment scale correlations with cognitive content from the Skills, Possessions, and Abilities instrument (SPA; see Table B.15). Two hundred and eleven participants completed the Hogan Judgment Report and the SPA. The sample included 95 males and 116 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.51 years (SD = 8.72). Although the SPA instrument included a broad range of content (e.g., computer usage and skill, number of various types of possessions owned, cultural knowledge, self-ratings of various talents), we were mainly interested in how the cognitive scales from the Hogan Judgment Report (i.e., Verbal and Numerical Information Processing) related to assessments of verbal and mathematical ability in the SPA. BRI. Finally, we review judgment scale correlations with cognitive content from the Behavioral Report Inventory (BRI; see Table B.16). Two hundred and five participants completed the Hogan Judgment Report and the BRI. The sample included 93 males and 112 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.32 years (SD = 8.63). Although the BRI included a broad range of content (e.g., 400 activity descriptions, 49 eating habits and practices, 24 symptoms of depression, 31 kinds of dissociative experiences), we were mainly interested in how the cognitive scales from the

3. VALIDITY

31

Hogan Judgment Report (i.e., Verbal and Numerical Information Processing) related to similar content on cognitive activities included in the BRI. 3.1.3 Results of Scale to Scale Correlates In the following section, we present selected statistically significant correlations between Hogan Judgment Report scales and those from the assessments previously described to illustrate relationships with behaviors that are conceptually aligned with the content of each scale. To examine full correlation matrices, please refer to Appendix B. Threat Avoidance vs. Reward Seeking. Table 3.1 provides selected significant correlations between the Threat Avoidance vs. Reward Seeking scale and scales from other assessments. Negative correlations indicate stronger relationships with Threat Avoidance whereas positive correlations indicate stronger relationships with Reward Seeking. Table 3.1 Construct Validity Evidence for Threat Avoidance vs. Reward Seeking Scale Assessment Scale Threat Avoidance vs. Reward Seeking HDS Cautious -.35** MVPI Security -.57** IPIP Conscientiousness -.31** TCI Fear of Uncertainty -.50** MPQ Harm Avoidance -.36** PRS Fearfulness -.44** HPI Ambition .34** Sociability .39** Inquisitive .57** HDS Mischievous .42** Colorful .47** MVPI Power .27** CPI Social Presence .39** Capacity for Status .36** NEO PI-R Openness/Actions .39** 16PF Openness to Change .36** CISS Adventuring Interests .31** Adventuring Skills .33** JPI-R Risk Taking .57** 6 FPQ Change Facet Scale .37** Openness to Experience .34** PRS Openness to Experience .38** Unconventionality .34** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Threat Avoidance; Positive correlations indicate relationships with Reward Seeking. Across assessments, four themes emerge. First, individuals who are open to change and new experiences tend to seek rewards when making decisions. Significant positive correlations between Reward Seeking and HPI Inquisitive (.57), NEO PI-R Openness/Actions (.39), 16PF Openness to Change (.36), CISS Adventuring Interests (.31) and Skills (.33), 6FPQ Change (.37) and Openness to Experience (.34), and PRS Openness to Experience (.38) and Unconventionality (.34) scales underscore these tendencies. Conversely, persons who fear change tend to adopt a threat avoidant perspective when making decisions.

3. VALIDITY

32

Significant negative correlations with HDS Cautious (-.35), MVPI Security (-.57), TCI Fear of Uncertainty (-.50), MPQ Harm Avoidance (-.36), and PRS Fearfulness (-.44) scales suggest that decision-makers who are open to novel experience tend to be more tolerant of uncertainty and driven by potential rewards, whereas individuals who are unassertive and fear the unknown tend to focus on potential risks. Second, individuals who enjoy power and status tend to adopt a reward-based goal orientation when making decisions. Significant positive correlations with HPI Ambition (.34), MVPI Power (.27), and CPI Capacity for Status (.36) indicate that motivation to achieve, influence, and obtain status may drive decisions that focus on rewards. Third, reward-seeking decision-makers also appear to be sociable and attention seeking, as evidenced by significant positive correlations with HPI Sociability (.39), HDS Colorful (.47), and CPI Social Presence (.39). These results indicate that individuals who enjoy social interaction may focus on rewards associated with potential public recognition. Fourth, individuals who are willing to take risks also tend to seek rewards. Significant positive correlations with HDS Mischievous (.42), JPI-R Risk Taking (.57), and the significant negative correlation with IPIP Conscientiousness (-.31), suggest that risk-takers tend to focus on rewards, whereas conscientious individuals focus on potential losses. Tactical vs. Strategic Thinking. Table 3.2 provides selected significant correlations between the Tactical vs. Strategic Thinking scale and scales from other assessments. Negative correlations indicate relationships with Tactical Thinking, whereas positive correlations indicate relationships with Strategic Thinking.

3. VALIDITY

33

Table 3.2 Construct Validity Evidence for Tactical vs. Strategic Thinking Scale Assessment Scale Tactical vs. Strategic Thinking HDS Skeptical -.26** Cautious -.62** Reserved -.31** NEO PI-R Self-Consciousness -.42** 16PF Apprehension -.27** MVPI Security -.28** HPI Ambition .66** HDS Colorful .44** MVPI Power .41** CPI Dominance .69** Leadership .57** NEO PI-R Extraversion/Assertiveness .67** Conscientiousness/Achievement Striving .40** IPIP Extraversion .47** 16PF Social Boldness .59** CISS Influencing Interests .46** Influencing Skills .47** TCI Resourcefulness .48** MPQ Social Potency .62** JPI-R Social Confidence .66** 6FPQ Extraversion .58** Dominance .52** PRS Social Boldness .69** Extraversion .55** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Tactical Thinking; Positive correlations indicate relationships with Strategic Thinking. Three themes are associated with this scale. First, people who are ambitious and driven to achieve tend to focus on larger, future-oriented issues. Significant positive correlations with HPI Ambition (.66), MVPI Power (.41), CPI Dominance (.69) and Leadership (.57), NEO PI-R Conscientiousness/Achievement Striving (.40), 16PF Social Boldness (.59), CISS Influencing Interests (.46) and Skills (.47), 6FPQ Dominance (.52), and PRS Social Boldness (.69) scales reflect this tendency. Second, individuals who are skilled at and enjoy social interaction tend to focus on big-picture issues over the long term. These tendencies are borne out by significant positive correlations with HDS Colorful (.44), NEO PI-R Extraversion/Assertiveness (.67), IPIP Extraversion (.47), MPQ Social Potency (.62), JPI-R Social Confidence (.66), 6FPQ Extraversion (.58), and PRS Extraversion (.55). Third, individuals who tend to be mistrustful, cautious, and apprehensive tend to take a practical and detailed approach to decision-making. Significant negative correlations with HDS Cautious (-.62), NEO PI-R Self-Consciousness (-.42), 16PF Apprehension (-.27), and MVPI Security (-.28) suggest that such individuals cling to relevant details as a means of coping with mistrust or fear. Conversely, the significant positive correlation with TCI Resourcefulness (.48) suggests that strategic thinkers are willing to make decisions and able to leverage available resources to inform those decisions.

3. VALIDITY

34

Data-Driven vs. Intuitive Decisions. Table 3.3 provides selected significant correlations between the Data-Driven vs. Intuitive Decisions scale and scales from other assessments. Negative correlations indicate relationships with Data-Driven Decisions, whereas positive correlations indicate relationships with Intuitive Decisions. Table 3.3 Construct Validity Evidence for Data-Driven vs. Intuit ive Decisions Scale Assessment Scale Data-Driven vs. Intuitive Decisions HPI Inquisitive -.47** Learning Approach -.33** MVPI Scientific -.67** IPIP Intellect -.29** CISS Analyzing Interests -.57** Analyzing Skills -.46** PRS Inquisitiveness -.52** CPI Intellectual Efficiency .29** MPQ Traditionalism .25** 16PF Social-Boldness .20** JPI-R Cooperativeness .21** Traditional Values .32** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Data-Driven Decisions; Positive correlations indicate relationships with Intuitive Decisions. Two themes emerge from these results for this scale. First, individuals who are naturally curious and analytical tend to make decisions based on a careful review of relevant data. Significant correlations with HPI Learning Approach (-.33), MVPI Scientific (-.67), IPIP Intellect (-.29), CISS Analyzing Interests (-.57) and Skills (-.46), and PRS Inquisitiveness (.-.52) scales reflect these tendencies. Decision-makers who like to learn and remain current with industry trends, and who are objective and analytical prefer to make decisions based on data. Second, socially bold leaders and those who value the past tend to make decisions based on career experience and intuition. Positive correlations with 16PF Social-Boldness (.20) and traditional value scales from the MPQ (.25) and JPI-R (.32) underscore this fact. The correlation with CPI Intellectual Efficiency (.29) provides additional evidence that intuitive decision-makers prefer to reach quick decisions guided by previous experience. Defensive vs. Cool-Headed. Table 3.4 presents selected significant correlations between the Defensive vs. Cool-Headed scale and those from other assessments. Negative correlations indicate Defensive reactions to negative feedback, whereas positive correlations indicate Cool-Headed reactions to feedback.

3. VALIDITY

35

Table 3.4 Construct Validity Evidence for Defensive vs. Cool-Headed Scale Assessment Scale Defensive vs. Cool-Headed HDS Excitable -.47** Skeptical -.45** NEO PI-R Neuroticism -.49** MPQ Stress Reaction -.51** JPI-R Anxiety -.49** HPI Adjustment .76** Prudence .48** CPI Well-Being .53** Self-Control .47** Amicability .48** Work Orientation .46** IPIP Emotional Stability .46** 16PF Emotional Stability .38** 6FPQ Even-Tempered .59** PRS Patience .54** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Defensive reactions; Positive correlations indicate relationships with Cool-Headed reactions.

Three themes are of note with this scale. First, emotionally stable individuals tend to be cool-headed when receiving feedback about their mistakes. This is seen in the positive correlations with HPI Adjustment (.76), IPIP Emotional Stability (.46), 16PF Emotional Stability (.38), 6FPQ Even-Tempered (.59), and PRS Patience (.54) scales. Conversely, individuals who are less emotionally poised tend to take a more defensive posture in reacting to feedback about their mistakes. Negative correlations with HDS Excitable (-.47), NEO PI-R Neuroticism (-.49), MPQ Stress Reaction (-.51), and JPI-R Anxiety (-.49) scales reflect these tendencies. Second, individuals who are dependable, self-controlled, and focused on the task at hand tend to remain calm in the face of negative feedback, as evidenced by positive correlations with HPI Prudence (.48), CPI Self-Control (.47), Amicability (.48), and Work Orientation (.46) scales. Because these individuals are normally disposed to take responsibility for their actions, they remain calm when receiving negative feedback because they remain focused on the work and how to make future improvements. Third, individuals who tend to be critical of themselves and others naturally become defensive when given negative feedback, as seen by the negative correlation with HDS Skeptical (-.45). It is possible that these individuals are more concerned with finding flaws in the feedback than with identifying errors in their decision-making, which may present obstacles to learning from their mistakes. Denial vs. Acceptance. Table 3.5 contains significant correlations between the Denial vs. Acceptance scale and those from other assessments. Negative correlations indicate Denial, whereas positive correlations indicate Acceptance.

3. VALIDITY

36

Table 3.5 Construct Validity Evidence for Denial vs. Acceptance Scale Assessment Scale Denial vs. Acceptance HPI Ambition -.58** HDS Bold -.55** MVPI Power -.58** Recognition -.41** CPI Dominance -.53** NEO PI-R Extraversion/Assertiveness -.52** Conscientiousness/Achievement Striving -.45** IPIP Extraversion -.30** 16PF Dominance -.43** Social Boldness -.37** CISS Influencing Interests -.35** Influencing Skills -.37** MPQ Social Potency -.57** Achievement -.36** JPI-R Social Confidence -.49** 6FPQ Extraversion -.50** PRS Extraversion -.42** HDS Cautious .44** NEO PI-R Self-Consciousness .22** Vulnerability .25** Agreeableness .29** Compliance .25** Modesty .35** 6FPQ Agreeableness .19** Good-Natured .21** PRS HEXACO Modesty .41** HEXACO Flexibility .21** HEXACO Honesty-Humility .25** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Denial; Positive correlations indicate relationships with Acceptance. Three themes emerge regarding this scale. First, ambitious individuals who value power and status tend to deny negative feedback. Negative correlations with HPI Ambition (-.58), HDS Bold (-.55), MVPI Power (-.58) and Recognition (-.41), CPI Dominance (-.53), NEO PI-R Conscientiousness/Achievement Striving (-.45), 16PF Dominance (-.43), CISS Influencing Interests (-.35) and Skills (-.37), and MPQ Achievement (-.36) reflect these tendencies. These results suggest that power seeking individuals tend to resist criticism, perhaps because they see it as a challenge to their authority. Outgoing and highly social individuals also tend to deny negative feedback. Negative correlations with NEO PI-R Extraversion/Assertiveness (-.52), IPIP Extraversion (-.30), 16PF Social Boldness (-.37), MPQ Social Potency (-.57), JPI-R Social Confidence (-.49), and Extraversion scales from the 6FPQ (-.50) and PRS (-.42) suggest that socially skilled individuals may view negative feedback as an assault on their social competence. Third, individuals who are cautious, modest, and agreeable tend to accept negative feedback about their mistakes. Significant positive correlations with HDS Cautious (.44), NEO PI-R Compliance (.25), Agreeableness scales from the NEO PI-R (.29) and 6FPQ (.19),

3. VALIDITY

37

and Modesty scales from the NEO PI-R (.35) and PRS (.41) suggest that humble and agreeable individuals are more likely to accept negative feedback than those who are bold and socially dominant. Superficial vs. Genuine Engagement. Table 3.6 provides selected correlations between the Superficial vs. Genuine Engagement scale and those from other assessments. Negative correlations indicate Superficial Engagement, whereas positive correlations indicate Genuine Engagement. Table 3.6 Construct Validity Evidence for Superficial vs. Genuine Engagement Scale Assessment Scale Superficial vs. Genuine Engagement HDS Excitable -.30** Cautious -.42** Leisurely -.32** MVPI Security -.26** NEO PI-R Neuroticism -.34** Self-Consciousness -.41** 16PF Apprehension -.34** JPI-R Cooperativeness -.33** MPQ Stress Reaction -.30** HPI Ambition .49** Inquisitive .31** Colorful .40** CPI Tough-Mindedness .46** Leadership .48** TCI Resourcefulness .45** MPQ Social Potency .26** JPI-R Social Confidence .37** 6FPQ Dominance .38** Extraversion .35** PRS Social Boldness .39** Note. ** Correlation is significant at .01 level; Negative correlations indicate relationships with Superficial Engagement; Positive correlations indicate relationships with Genuine Engagement. Across assessments, four themes emerge. First, achievement oriented individuals tend to genuinely engage in feedback. Positive correlations with HPI Ambition (.49), CPI Tough-Mindedness (.46) and Leadership (.48), TCI Resourcefulness (.45), and 6FPQ Dominance (.38) scales support these tendencies, and suggest that decision-makers who are competitive, leader-like, and willing to make tough decisions are willing to take on negative feedback in order to improve future performance. Second, individuals who are socially skilled are also willing to genuinely engage in negative feedback. Positive correlations with HDS Colorful (.40), MPQ Social Potency (.26), JPI-R Social Confidence (.37), 6FPQ Extraversion (.35), and PRS Social Boldness (.39) scales indicate that individuals who are adept at interacting with others are willing to listen to feedback in order to learn from their past mistakes. Third, individuals who are emotionally volatile tend to superficially engage in feedback to maintain positive social impressions and avoid a genuine confrontation with their problems.

3. VALIDITY

38

Negative correlations with HDS Excitable (-.30), NEO PI-R Neuroticism (-.34), and MPQ Stress Reaction (-.30) suggest that these individuals avoid deeper exploration of their failures because doing so is threatening. Finally, individuals who value predictability and order or are cautious, apprehensive, and cooperative may also superficially engage in negative feedback. Negative correlations with HDS Cautious (-.42), NEO PI-R Self-Consciousness (-.41), 16PF Apprehension (-.34), JPI-R Cooperativeness (-.33), and MVPI Security (-.26) underscore these points. In general, individuals who lack poise are more likely to engage superficially compared to those who remain calm and confident. Openness to Feedback and Coaching. Table 3.7 provides selected correlations between the overall Openness to Feedback and Coaching scale from the Hogan Judgment Report and scales from other assessments. Table 3.7 Construct Validity Evidence for Openness to Feedback and Coaching Scale Assessment Scale Openness to Feedback and Coaching HDS Excitable -.40** Skeptical -.41** NEO PI-R Neuroticism -.44** Angry Hostility -.47** MPQ Stress Reaction -.43** PRS HEXACO Anxiety -.42** MVPI Hedonism -.42** Recognition -.45** JPI-R Anxiety -.50** HPI Adjustment .56** CPI Self-Control .49** Well-Being .51** Tolerance .44** Ego-Integration .44** IPIP Emotional Stability .50** 6FPQ Even-Tempered .46** Note. ** Correlation is significant at .01 level. Three themes emerge with regard to this scale. First, emotionally stable individuals seem most able to profit from coaching and learning from that process. Positive correlations with HPI Adjustment (.56), CPI Self-Control (.49), IPIP Emotional Stability (.50), and 6FPQ Even-Tempered (.46) scales reflect the beneficial nature of these tendencies. Second and conversely, highly emotional individuals seem least receptive to feedback and coaching around their past mistakes. Negative correlations with HDS Excitable (-.40), NEO PI-R Neuroticism (-.44) and Angry Hostility (-.47), MPQ Stress Reaction (-.43), HEXACO Anxiety from the PRS (-.42), and JPI-R Anxiety (-.50) underscore this point. Third, individuals who do not place value in others, or who place excess value on leisure and personal recognition, may encounter difficulties in remaining receptive to feedback and coaching. Negative correlations with HDS Skeptical (-.41) and MVPI Hedonism (-.42) and

3. VALIDITY

39

Recognition (-.45) reflect the fact that trust in others, humility, and a business-like approach are often needed to benefit from feedback and coaching. Verbal Information Processing. Table 3.8 provides selected significant correlations between scores on the Verbal Information Processing scale from the Hogan Judgment Report and cognitive content from other assessments. Table 3.8 Construct Validity Evidence for Verbal Information Processing Scale

Assessment Scale Verbal Informational Processing SPA Verbal Ability .16* BRI Reading Cluster .15* Books Cluster .15* HPI Learning Approach .32** HBRI Tactical Thinking .45** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Across assessments, Verbal Information Processing scale scores show significant positive correlations with SPA Verbal Ability (.16), and BRI Reading (.15) and Books (.15) Clusters. The strongest noted correlation for this scale is with HBRI Tactical Thinking (.45), suggesting that individuals who quickly process and analyze verbal information also tend to focus on practical concerns and extract relevant information from verbal cues to make informed decisions. Finally, the Verbal Information Processing scale shows a strong positive correlation with HPI Learning Approach (.32), suggesting that individuals who are adept at processing verbal information also tend to value education and staying up-to-date with developments in their field. As a whole, these results suggest the verbal information processing scale assesses an individual’s ability to process detailed information presented in text and use that information in decision-making. Numerical Information Processing. Table 3.9 provides selected significant correlations between scores on the Numerical Information Processing scale and cognitive content from other assessments. Table 3.9 Construct Validity Evidence for Numerical Information Processing Scale

Assessment Scale Numerical Informational Processing SPA Mathematical Ability .20** BRI Computing Cluster .25** JPI-R Complexity .22** HPI Inquisitive .22* Learning Approach .26** HBRI Strategic Thinking .41** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

Across assessments, Numerical Information Processing scale scores show significant positive correlations with SPA Mathematical Ability (.20), BRI Computing (.25), and JPI-R Complexity (.22) under the Analytical cluster. These results suggest that individuals who

3. VALIDITY

40

enjoy mathematical computations can also quickly process and analyze numerical data when making decisions. Additionally, scores on the Numerical Information Processing scale correlate significantly with HPI Inquisitive (.22) and Learning Approach (.26), suggesting that individuals who can quickly recognize patterns in numerical data also tend to be curious and critical thinkers who prefer to keep up-to-date with developments in their industries. Finally, the strongest noted correlation for Numerical Information Processing scale scores is with HBRI Strategic Thinking (.41), suggesting that individuals who possess superior analytical skills also tend to see the big picture in their analyses of numerical data and leverage those analyses to make strategic decisions with long-term implications. 3.2 Correlations with Others’ Descriptions One of the most important sources of information for interpreting personality scales is correlations between scale scores and descriptions of a person as rated by observers. We obtained this information from the Eugene – Springfield Community Sample (ESCS; Goldberg, 2008) using persons who completed the Hogan Judgment Assessment and had previously been rated by observers on standardized checklists. In the following section, we provide selected correlations between judgment scales and these observer ratings. For full correlation matrices with these instruments, please refer to Appendix B. 3.2.1 Procedure 3.2.2 Samples and Instruments Adjective Descriptions and Personality Phrases. As part of Goldberg’s (2008) longitudinal community research, respondents and observers (e.g., significant others, spouses, friends, acquaintances, coworkers) completed the Self/Peer Inventories, which includes 88 items taken from Saucier’s (1994) 40-item Big-Five “Mini-Markers,” the 44-item Big-Five Inventory (John & Srivastava, 1999; Benet-Martinez & John, 1998), and two additional items in each inventory measuring physical attractiveness. In this survey, respondents described how well each adjective or phrase described either themselves or the target individual using a 5-point Likert scale ranging from 1 (Extremely Inaccurate) to 5 (Extremely Accurate). Each participant, and up to four observers of each participant, completed these 88 items. The sample of 196 participants providing self-ratings included 87 males and 109 females. Ages of subjects ranged from 21 years to 72 years with a mean of 45.45 years (SD = 8.72). Observers also responded to items assessing how and how well they knew the target, how much they liked the target, and basic demographic questions on gender and age. The sample of 538 respondents providing observer ratings included 208 males and 330 females. Ages of observers ranged from 7 years to 89 years with a mean of 41.50 years (SD = 16.24). Observers were split between spouses and other relatives (N = 300) compared with friends, coworkers, acquaintances, and significant others (N = 207), with 31 observers

3. VALIDITY

41

not indicating their relationship to the target. Most observers indicated knowing the target “well” or “very well” (N = 522), and most indicated that they “liked” the target or liked the target “very much” (N = 520). For each of the 88 items, observer ratings were pooled into a composite by calculating a mean response across all observers. We used these mean responses (N = 196) as the basis for calculating correlations between observer ratings and the Hogan Judgment Report scales. 3.2.3 Results of Scale and Observer Description Correlates The following section presents selected significant correlations with observer ratings that align with our theory of each scale’s content. Tables B.17 and B.18 in Appendix B provide complete correlation matrices for adjective and phrase descriptors, respectively. Threat Avoidance vs. Reward Seeking. Table 3.10 provides the five strongest negative correlations and the five strongest positive correlations between Threat Avoidance vs. Reward Seeking scale scores and observer ratings for descriptive adjectives and phrases. Table 3.10 Correlations with Threat Avoidance vs. Reward Seeking and Observer Ratings Descriptor Correlation Descriptor Correlation Uncreative -.26** Complex .21** Bashful -.19** Imaginative .19** Sympathetic -.16* Disorganized .16* Organized -.14* Philosophical .16* Efficient -.13 Energetic .15* Prefers routine work -.32** Full of energy .22** Likes to cooperate with others -.19** Curious about many different things .22** Considerate and kind to almost everyone -.17* Has an active imagination .21** Helpful and unselfish with others -.15* Inventive .16* Perseveres until the task is finished -.15* Original/comes up with new ideas .15* Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Threat Avoidance; Positive correlations indicate relationships with Reward Seeking. Results suggest that Threat Avoidant individuals seem uncreative (-.26) and preferring routine work (-.32), whereas Reward Seeking individuals seem complex (.21), energetic (.22), naturally curious (.22), and imaginative (.21). Tactical vs. Strategic Thinking. Table 3.11 provides the five strongest negative and positive correlations between Tactical vs. Strategic Thinking scale scores and observer ratings.

3. VALIDITY

42

Table 3.11 Correlations with Tactical vs. Strategic Thinking and Observer Ratings Descriptor Correlation Descriptor Correlation Bashful -.37** Bold .42** Shy -.30** Extraverted .31** Quiet -.27** Complex .26** Withdrawn -.25** Energetic .24** Unintellectual -.21** Intellectual .24** Sometimes shy/inhibited -.36** Has an assertive personality .39** Prefers routine work -.34** Outgoing/sociable .35** Is reserved -.28** Generates a lot of enthusiasm .29** Tends to be quiet -.28** Has an active imagination .24** Tends to be lazy -.23** Ingenious/deep thinker .24** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Tactical Thinking; Positive correlations indicate relationships with Strategic Thinking. Results suggest that Tactical Thinkers are described as bashful (-.37), inhibited (-.36), preferring routine work (-.34), and shy (-.30), whereas Strategic Thinkers are described as bold (.42), assertive (.39), outgoing (.35), and extraverted (.31). Data-Driven vs. Intuitive Decisions. Table 3.12 provides the five strongest negative and positive correlations between Data-Driven vs. Intuitive Decisions scale scores and observer ratings. Table 3.12 Correlations with Data-Driven vs. Intuit ive Decisions and Observer Ratings Descriptor Correlation Descriptor Correlation Complex -.31** Good-Looking .20** Quiet -.30** Talkative .17* Practical -.18* Unintellectual .17* Intellectual -.18* Extraverted .14* Systematic -.14* Bold .14* Tends to be quiet -.28** Physically attractive .22** Curious about many different things -.23** Outgoing/sociable .16* Likes to reflect/play with ideas -.22** Generates a lot of enthusiasm .13 Ingenious/deep thinker -.21** Gets nervous easily .13 Is reserved -.19** Has an assertive personality .12 Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Data-Driven Decisions; Positive correlations indicate relationships with Intuitive Decisions. Results suggest that individuals who base their decisions on data seem complex (-.31) and quiet (-.30), whereas intuitive decision-makers seem physically attractive (.22), talkative (.17), and unintellectual (.17). Defensive vs. Cool-Headed. Table 3.13 provides the five strongest negative and positive correlations between Defensive vs. Cool-Headed scale scores and observer ratings.

3. VALIDITY

43

Table 3.13 Correlations with Defensive vs. Cool-Headed Reactions and Observer Ratings Descriptor Correlation Descriptor Correlation Temperamental -.30** Relaxed .30** Moody -.26** Cooperative .22** Jealous -.24** Kind .19** Harsh -.23** Unenvious .19** Touchy -.23** Practical .11 Can be tense -.32** Emotionally stable/not easily upset .38** Finds fault with others -.28** Relaxed/handles stress well .33** Is depressed/blue -.28** Remains calm in tense situations .24** Gets nervous easily -.28** Makes plans and follows through .21** Worries a lot -.27** Likes to cooperate with others .19** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Defensive reactions; Positive correlations indicate relationships with Cool-Headed reactions. Results suggest that Defensive individuals seem tense (-.32) and temperamental (-.30), whereas Cool-Headed individuals seem not easily upset (.38), adept at handling stress (.33), and relaxed (.30). Denial vs. Acceptance. Table 3.14 provides the five strongest negative and positive correlations between Denial vs. Acceptance scale scores and observer ratings. Table 3.14 Correlations with Denial vs. Acceptance and Observer Ratings Descriptor Correlation Descriptor Correlation Bold -.29** Bashful .24** Complex -.28** Uncreative .19** Intellectual -.25** Unintellectual .18* Energetic -.20** Shy .15* Extraverted -.20** Withdrawn .12 Ingenious/deep thinker -.28** Prefers routine work .29** Has an assertive personality -.27** Sometimes shy/inhibited .24** Original/comes up with new ideas -.25** Tends to be lazy .21** Inventive -.23** Is reserved .15* Full of energy -.22** Has a forgiving nature .14* Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Denial; Positive correlations indicate relationships with Acceptance. Results suggest that people who deny the validity of feedback seem bold (-.29), complex (-.28), and assertive (-.27), whereas individuals who accept feedback seem to prefer routine work (.29), and may be bashful (.24) and sometimes inhibited (.24). Superficial vs. Genuine Engagement. Table 3.15 provides the five strongest negative and positive correlations between Superficial vs. Genuine Engagement scale scores and observer ratings.

3. VALIDITY

44

Table 3.15 Correlations with Superficial vs. Genuine Engagement and Observer Ratings Descriptor Correlation Descriptor Correlation Withdrawn -.28** Bold .28** Bashful -.26** Energetic .18* Fretful -.21** Practical .15* Shy -.21** Extraverted .13 Moody -.15* Complex .12 Gets nervous easily -.28** Outgoing/sociable .24** Is depressed/blue -.26** Has an assertive personality .23** Sometimes shy/inhibited -.21** Full of energy .22** Is reserved -.19** Relaxed/handles stress well .20** Can be moody -.18* Remains calm in tense situations .19** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level; Negative correlations indicate relationships with Superficial Engagement; Positive correlations indicate relationships with Genuine Engagement. Results suggest that Superficially Engaged individuals seem nervous (-.28), withdrawn (-.28), and bashful (-.26), whereas Genuinely Engaged individuals seem bold (.28), outgoing (.24), assertive (.23), energetic (.22), and adept at handling stress (.20). Openness to Feedback and Coaching. Table 3.16 provides the five strongest negative and positive correlations between Openness to Feedback and Coaching scale scores and observer ratings. Table 3.16 Correlations with Openness to Feedback & Coaching and Observer Ratings Descriptor Correlation Descriptor Correlation Temperamental -.30** Relaxed .25** Jealous -.25** Cooperative .21** Harsh -.25** Unenvious .14 Fretful -.24** Practical .13 Touchy -.24** Warm .13 Is depressed/blue -.32** Emotionally stable/not easily upset .33** Gets nervous easily -.30** Relaxed/handles stress well .23** Finds fault with others -.26** Remains calm in tense situations .22** Worries a lot -.25** Has a forgiving nature .19** Can be tense -.25** Likes to cooperate with others .18** Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Results suggest that individuals who resist coaching seem depressed (-.32), temperamental (-.30), and nervous (-.30), whereas individuals who are generally receptive to feedback and coaching seem emotionally stable (.33), relaxed (.25), adept at handling stress (.23), and calm in tense situations (.22). Verbal Information Processing. Table 3.17 provides the five strongest negative and positive correlations between Verbal Information Processing scale scores and observer ratings.

3. VALIDITY

45

Table 3.17 Correlations with Verbal Information Processing and Observer Ratings Descriptor Correlation Descriptor Correlation Unintellectual -.23** Organized .21** Disorganized -.22** Efficient .17* Talkative -.19** Deep .17* Temperamental -.13 Intellectual .17* Touchy -.11 Systematic .15* Easily distracted -.18* Does things efficiently .20** Tends to be disorganized -.17* Original/comes up with new ideas .15* Can be somewhat careless -.16* Ingenious/deep thinker .14 Outgoing/sociable -.12 Does a thorough job .13 Starts quarrels with others -.10 Sophisticated in art, music, literature .12 Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level. As seen in Table 3.17, correlations between Verbal Information Processing scale scores and observer ratings suggest that individuals who struggle to process verbal information seem unintellectual (-.23), disorganized (-.22), and talkative (-.19), whereas those who can quickly process this information seem organized (.21), efficient (.20), and intellectual (.17). Numerical Information Processing. Table 3.18 provides the five strongest negative and positive correlations between Numerical Information Processing scale scores and observer ratings. Table 3.18 Correlations with Numerical Information Processing and Observer Ratings Descriptor Correlation Descriptor Correlation Unintellectual -.19** Intellectual .22** Talkative -.12 Practical .16* Uncreative -.12 Deep .13 Jealous -.11 Systematic .13 Fretful -.11 Unenvious .11 Tends to be disorganized -.10 Ingenious/deep thinker .18** Is depressed/blue -.08 Sophisticated in art, music, literature .15* Gets nervous easily -.08 Makes plans and follows through .15* Easily distracted -.07 Likes to reflect/play with ideas .13 Has few artistic interests -.07 Original/comes up with new ideas .12 Note. * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Results suggest that individuals who struggle to process quantitative data seem unintellectual (-.19) and uncreative (-.12), whereas those who can quickly process numerical information seem intellectual (.22) and deep thinking (.18). Taken together, these results provide additional support for the construct validity of the Hogan Judgment Report scales. More importantly, these results indicate that an individual’s natural decision-making tendencies, reactions to feedback, receiving feedback and coaching, and processing verbal and numerical information are reflected in that person’s reputation (i.e., how others describe them).

3. VALIDITY

46

3.3 Criterion-Related Validity Aguinis, Henle, and Ostroff (2001) described criterion-related validity in terms of the relationship between a predictor and some criterion measure (e.g., job performance), with the goal of answering the basic question, “How accurate are the assessment scores in predicting criterion performance?” The Uniform Guidelines state that “evidence of the validity of a test…by a criterion-related validity study should consist of empirical data demonstrating that the [test] is predictive of or significantly correlated with important elements of job performance” (29 C.F.R. § § 1607.5 (B)). Although many organizational and logistical constraints limit the usefulness of criterion-related validity studies (McPhail, 2007), the Uniform Guidelines and Principles suggest considering this approach when a) there is an adequate, representative sample of job incumbents willing to participate, and b) development of reliable, unbiased measures of job performance is possible. The Principles also recommends using a relevant criterion measure, one that “reflects the relative standing of employees with respect to important work behavior(s) or outcome measures(s)” (p. 14). 3.3.1 Procedure We are collecting criterion-related validation data for the Hogan Judgment Report scales as part of global research studies, using online, unproctored Internet testing. Specifically, we have offered complimentary Hogan Judgment Reports to individuals in professional and managerial roles across the globe in exchange for their completion of the Hogan Judgment Assessment and their supervisor’s completion of a conceptually-aligned performance rating form described below. No participants took the assessment as part of high stakes testing where results impacted hiring, promotion, or other personnel decisions. 3.3.2 Sample and Instrument Judgment-Related Performance Rating Form. To collect conceptually-aligned job performance criterion data, we created a judgment-related Performance Rating Form (PRF) based on our theory of each scale’s content. The PRF includes 34 questions and is divided into four sections: (a) overall performance, (b) decision-making descriptions, (c) cognitive ability, and (d) open-ended comments. For the first section, we ask supervisors to rate (a) the employee’s overall job performance using a scale ranging from 1 (Not Effective) to 5 (Exceptional), (b) whether the employee is a model for new employees’ work behavior using a scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree), (c) what their response would be if the employee was thinking of leaving the organization, with options ranging from 1 (Do Nothing) to 3 (Make a Significant Effort to Keep Him/Her), and (d) how often they agree with the employee’s decisions, using a scale ranging from 1 (Rarely) to 4 (Always). Remaining questions in this section assess whether the employee (a) exemplifies organizational values, (b) makes good decisions at work, and (c) usually seeks out coaching or feedback on performance. Rating scales for these remaining questions use the Strongly Disagree – Strongly Agree scale previously described.

3. VALIDITY

47

To examine decision-making, we present 10 pairs of descriptors (e.g., This employee tends to focus on the details vs. This employee tends to focus on the big picture) and ask supervisors to (a) decide which descriptor best describes the employee, and (b) whether the employee shows either a Moderate Preference or Strong Preference for the descriptor that accurately describes them. Next, we ask supervisors to rate the employee on 10 items related to general cognitive ability in the context of making work-related decisions. Example items ask whether the employee gathers essential information needed to make decisions, seeks out and considers alternative decisions, makes good decisions under ambiguous circumstances, and makes good decisions under stress. Supervisors respond to these items using a scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Finally, we provide supervisors with an opportunity to provide open-ended comments about their employee’s work-related decision-making. Specifically, we ask supervisors to describe (a) the employee’s three biggest strengths related to decision-making, and (b) their three biggest challenges to making work-related decisions. At present, we have collected data from approximately one hundred individuals who have completed the Hogan Judgment Assessment and been rated by their supervisors on the Performance Rating Form. Although this sample would allow us to report favorable preliminary findings, our current sample size does not allow us to draw generalizable conclusions based on statistically stable results. As such, we will report these findings as soon as we have obtained data from a sufficiently large and representative sample.

4. INTERPRETATION & USES

48

4. Interpretation and Uses The Hogan Judgment Report is designed primarily for leadership development, training and coaching focused on decision-making. We assume that making people aware of their typical decision-making tendencies should help them make better future decisions. The Judgment Report is suitable for pre-hire assessment when the goal is to ensure that a candidate’s decision-making and thinking styles are a good fit with the demands of a role, team, or organization. The Judgment Report provides feedback and developmental tips in four areas: (a) numerical and verbal information processing, (b) natural pre-decision tendencies, (c) reactions to feedback about failed decisions, and (d) overall coachability. The report can help create strategic self-awareness concerning decision-making and responses to feedback regarding bad decisions, both of which are critical to job performance, career success, and leadership effectiveness. Below we describe characteristics associated with different scores for each of the Judgment scales. These interpretive statements were derived from (a) descriptions provided by co-workers, and (b) demonstrated relationships between the scales and respondents’ scores on well-established personality, values, and ability measures. We also provide some developmental suggestions; these recommendations must be tailored to the specific needs of the individual and will be most effective when implemented in consultation with a trusted mentor or coach. 4.1 Information Processing People differ in speed of processing information, and some are faster than others. These differences are usually attributed to short-term memory capacity and attentional focus. Information processing capacity enables people to solve well-structured problems by identifying patterns, making connections, or thinking abstractly. There are two main domains of information processing – verbal and numerical. Verbal tests concern the ability to solve problems based on words or concepts. Numerical tests concern a person’s ability to solve problems based on numbers or abstract reasoning. The information processing section of the Hogan Judgment Report concerns both verbal and numerical information processing. Below, we provide interpretive content for scores in each of these sections. 4.1.1 Verbal Information Processing The verbal section evaluates the ability to process information presented with words or verbal content. Although the problems in this section require little previous knowledge or vocabulary, people who perform well should have stronger verbal ability. They tend to be interested in the humanities and are generally a good fit for careers that require using words to convey emotions and tell stories—e.g., literature, journalism, law, and education. Conversely, individuals with lower scores on this section typically require more time to process verbal information.


49

4.1.2 Numerical Information Processing The numerical information processing section evaluates the ability to process information presented only in numerical data. Although the problems in this section require only basic command of arithmetic, people who perform well should have stronger math skills and good logical reasoning ability. They tend to be interested in scientific fields and are generally a good fit for careers that require data-based problem solving—e.g., mathematics, engineering, accounting, and Information Technology. Conversely, individuals with lower scores on this section typically require more time to process numerical data. 4.1.3 Information Processing Styles As shown in Figure 4.1, people tend to fall into one of four groups depending on their information processing style. The distinction between qualitative and quantitative information processing styles is based on C.P. Snow’s (1959) notion of “two cultures”, namely Sciences vs. Humanities, and highlights the two different modes of information processing used in decision-making. We present interpretive and developmental content for each information processing style below. Figure 4.1 Information Processing Styles

4.1.3.1 Deliberate Individuals with average scores for both verbal and numerical information processing tend to analyze information in a deliberate manner. They take their time in making decisions to ensure that they understand the issues completely. They tend to be described by others as organized and systematic, and are often viewed as practical and detailed.


50

4.1.3.2 Quantitative Individuals with high scores for numerical, but average scores for verbal information processing enjoy analyzing mathematical problems and prefer to focus on facts rather than people. They are often good at translating data into guidelines for decision-making, and view life as a series of problems to be solved. Quantitative information processors often excel at evaluating factual issues, and prefer concrete challenges to ambiguously defined problems. They tend to be described as ingenious deep-thinkers, and are often viewed as sophisticated and good at following through. 4.1.3.3 Qualitative Persons with high scores for verbal, but average scores for numerical data processing, tend to be good at using language to analyze problems, communicate decisions, and express their emotions. This information processing style also facilitates understanding and interpreting others’ feelings and intentions. Qualitative information processors are more interested in people issues than abstract, logical, or mathematical problems. Others tend to describe them as deep, philosophical, and original thinkers who often generate new ideas. 4.1.3.4 Versati le Individuals with high scores for both numerical and verbal information processing are versatile information processors who can effectively use both numerical and verbal information for decision-making. They tend to do well in many fields. In general, they are insightful about both people issues and data-based problems. Others often describe them as intellectual thinkers who can process information and follow through on decisions. 4.2 Decision-Making Approaches Another key aspect of individual differences in judgment concerns how people approach decisions. Unlike information processing style, decision-making tendencies involve automatic, spontaneous, and subconscious tendencies, or dispositional strategies that people adopt when solving ill-defined or ambiguous problems. Because most of the problems we encounter at work don’t have an objectively correct solution and contain insufficient information to demand careful analysis, we normally rely on mental shortcuts – spontaneous decision heuristics that represent our default problem solving tendencies. People make business decisions using certain predictable tendencies: (a) threat avoidance vs. reward seeking; (b) tactical vs. strategic thinking; and (c) data-driven vs. intuitive decisions. Every individual can be described in terms of their placement along these three dimensions. For example, people with strong reward seeking tendencies are likely to approach decisions with an “eye on the prize” – they tend to evaluate options based on potential positive outcomes and select the approach that will lead to the biggest reward. On the other hand, people who fear failure tend to evaluate options based on potential negative consequences and avoid risky choices with the potential for bigger rewards. People with a more balanced approach consider both potential rewards and risks associated with each possible course of action.


51

The Hogan Judgment Report describes the extent to which a person relies on the tendencies defined by these scale pairs. Understanding these tendencies can: • Enhance an individual’s strategic self-awareness of his/her decision-making style • Help people identify resources to complement their natural decision-making style • Facilitate professional development by highlighting potential strengths and challenges

inherent in a person’s natural decision-making tendencies • Inform talent decisions (e.g., selection and placement) by identifying individuals whose

natural decision-making tendencies fit the requirements of a role • Facilitate coaching and training efforts by highlighting potential challenges that may arise

as the individual attempts to change and improve 4.2.1 Threat Avoidance vs. Reward Seeking People who are threat avoidant focus on the negative side of the risk-reward equation; they make decisions intended to minimize potential financial, legal, physical, material, personnel, or other losses. Conversely, people who are reward-seeking focus on the positive side of the risk-reward equation; they make decisions intended to maximize their potential rewards. In simple terms, threat-avoidance concerns being motivated by avoiding the “sticks”, whereas reward-seeking concerns being motivated by earning the “carrots”. Threat avoidant individuals try to anticipate future threats and potential risks. When evaluating options during decision-making, they focus on what could go wrong. They approach decision-making opportunities in a structured manner, and often follow a defined process to evaluate the situation and potential courses of action. As a result, others may see them as predictable and cautious in their problem-solving and decision-making. Although colleagues may see them as reliable and dependable, threat avoidant decision-makers may not solicit input in the decision-making process, because other viewpoints may be distracting. Threat avoidant people also struggle to influence others to support their decisions, and may seem overly concerned about unlikely or relatively insignificant problems. In many cases, they may choose the alternatives that include the fewest unknowns and are likely to result in positive outcomes even if the benefits are modest. Although others may not expect them to make radical or exciting decisions, they may be relied upon consistently to make solid, safe decisions. In contrast, reward seeking individuals stay alert for opportunities for gain and growth. When making decisions, reward seekers may emphasize the potential positive outcomes associated with various options. They tend to adopt a “big risk, big reward” attitude and seek to maximize the possible gains associated with every decision. They regard each decision point as a potential “big win”. As a result, they may choose alternatives with great potential even if the alternatives are obviously dangerous. Reward seekers often take a “dynamic approach”, using a process that can be adapted to explore any potentially advantageous paths discovered during decision-making. Reward seekers may also include others, leveraging connections to build support for decisions, including those that involve significant risk. They are often able to influence others to move outside their comfort zones and embrace risks in order to achieve rewards. Nonetheless, reward seekers may take unnecessary or even inadvisable risks to pursue potential rewards. They may ignore


52

processes and rules that impede their ability to make what they consider to be optimal decisions. Others tend to see them as interesting and engaging, but also prone to testing limits and taking risks without fully considering the consequences of their actions. Reward seekers may be impulsive and take action quickly. Although this may sometimes pay off, it will also lead to avoidable losses along with wins. Consequently, reward seekers may benefit from soliciting others’ input to see if they perceive the same opportunities. Reward seekers can benefit from considering potential risks as well as rewards. 4.2.1.1 Developmental Recommendations for Threat Avoidance • During decision-making, ask others what risks they perceive in the situation. Do they see

the same threats and potential negative side effects in each alternative? If so, how concerned are they about the threats?

• Frame your decision-making rationale in optimistic terms by starting with positives, then discussing potential hazards, and then finishing on a positive note.

• When next faced with an opportunity that seems too risky, consider the worst that might happen if you did take action. If the “worst case scenario” is relatively innocuous, you should practice taking risks and moving to action despite uncertainty.

• Create a process for evaluating risk, so that you have a consistent method for analyzing the pros and cons of situations, determining the most appropriate action to take, and evaluating the outcomes of that action.

• Identify past situations in which you took risks and they paid off. What situational factors help you feel comfortable (e.g., working collaboratively, working on lower stakes projects, working with a leader who supports risk taking)? Consider how you might replicate those factors in future situations.

4.2.1.2 Developmental Recommendations for Reward Seeking • Before making decisions, critically review your alternatives in terms of the amount of risk

involved. Identify options that can reduce potential unintended consequences associated with the decision. Small changes may reduce unnecessary risk or help you better define the unknowns in the situation.

• Use your team to help you identify the risks in a given decision. Your more cautious colleagues may see things that you don't, and their insights may help you better understand what could go wrong and how others may perceive your decision once it’s made.

• Identify situations in which you may have taken a "ready, fire, aim" approach. What factors lead you to move quickly without fully evaluating the circumstances? Do you tend to act quickly when you are bored? When you are under heavy pressure? When you are in routine situations? Identify checks-and-balances that you can use in these situations to ensure you properly consider your options before moving to action.

• For the next few months, keep a journal of your significant decisions. List both the pros and cons of your decision and ask your boss or a trusted associate to do the same. Once you have created a record, review each decision. How many times did you "win"? What risks or potential negative consequences did you fail to consider prior to making a


53

decision? How could your decision-making process be improved to increase the number of wins?

• There is a difference between seeking input from others and seeking their buy-in. Before making decisions, seek constructive feedback on your approach. Find ways to incorporate that feedback into your decision-making process. It may not change your final decision, but you can at least use it to improve your approach, inform how you manage the potential negative impact of the decision, or guide how you evaluate the decision after it’s made. Input from others may improve your process and decisions and demonstrate that you value their opinions and are not just trying to influence them to support you.

4.2.2 Tactical vs. Strategic Thinking Tactical thinkers approach decisions with a short-term, detail-oriented perspective. They focus on immediate concerns such as cost, implementation issues, and staffing problems and tend not to put these issues in a larger context. Conversely, strategic thinkers tend to put problems in context, and look for systematic or comprehensive solutions to problems rather than treating symptoms as they occur; they often initiate strategic change but may ignore more detailed issues such as cost and implementation. Tactical thinkers are skilled at identifying troublesome details and implementing programs. They approach decisions by identifying the specific goal that the decision is meant to achieve, then finding the most direct route to that outcome. Put another way, tactical thinkers focus on the journey rather than the destination. They often seem to prefer to let others define the goals rather than propose their own vision or strategy. Tactical thinkers tend to work backwards from the desired goal to identify the concrete steps needed to achieve that goal and mitigate the risk of overlooking critical details. Others see them as detail-oriented and methodical, although they may sometimes get lost in the details and lose sight of the broader implications of the decision. Tactical thinkers often rely on established practices and proven solutions to guide their decisions; this may lead others to view them as efficient decision-makers because they don’t spend time considering untenable approaches that have to eventually be abandoned. However, tactical thinking may lead to missed opportunities for more innovative or forward-thinking decisions. Tactical thinkers are usually well-equipped to make pragmatic decisions that require synthesizing detailed information. Additionally, they work well with more innovative problem-solvers, providing checks and balances to ensure that important operational factors are not overlooked in the decision-making process. Strategic thinkers tend to focus on identifying larger business trends and developing overarching business plans. When making decisions, they pursue broad, flexible objectives, and avoid defining concrete, narrow goals. They are often reluctant to choose specific paths to avoid closing the door on both known and unknown possibilities. They often rely on an overall strategy or vision to guide their decision-making, focusing on the destination more than the journey. As a result, others may view strategic thinkers as adaptable and able to follow multiple paths to make the best decision in a given situation. Strategic thinkers tend not to rely on standard processes or methodologies, preferring instead to let specific situations dictate their decision-making approach. During the decision-making process, they


54

may continuously respond to new information and changing circumstances and adapt their thinking to the changing decision landscape. Consequently, others may find it difficult to predict the strategic thinker’s decisions. Others may believe in a strategic thinker’s ability to set strategy and articulate a vision, but question his/her ability to consider the details entailed by their strategies. Strategic thinkers shine when they can develop a 30,000-foot view strategy. But when faced with routine decisions, they may become bored and take action without considering important nuances of the situation that should inform the decision. They may work well with tactical decision-makers who focus on the operational and logistical details that strategic thinkers ignore. 4.2.2.1 Developmental Recommendations for Tactical Thinkers • Focus on the global and macro-economic trends in your industry and connect them with

your organization’s strategy. Identify the ways in which the decisions you make relate to (a) your organization’s broader strategy, and (b) the conditions in your industry.

• Think of a time when you found it difficult to make an important decision in an ambiguous situation. What were the specific unknowns in the situation? How did you address these factors in order to make a decision? Consider your typical decision-making process and how you might adapt it to better manage unexpected challenges when they arise in the future. Identify techniques you might use to manage uncertainty so that it doesn’t impede your decision-making. For example, you might ask others to help you better define the unknowns in a given situation, or develop contingency plans so that your decision can be modified if new information comes to light.

• You may approach decisions by first defining the goal. Consider how the immediate goal might serve a larger mission or vision. Try to document how your immediate goal might contribute to the team or organization’s long-term objectives. Share this with a trusted colleague, supervisor, or mentor and ask them to fill in other connections and paths that might not have occurred to you.

• Look for ways your decisions might conflict with broader strategies and objectives. Consider how your present choices could conflict with some desired future state. Use this forward-thinking perspective to inform your decisions by both their short-term, known consequences and their potential future consequences. Try to balance these critical perspectives when making decisions.

• Identify key stake-holders who may take a more strategic approach to decision-making. Include them in your decision-making process, so that you benefit from their big-picture perspective while incorporating your own strengths around detail-orientation and day-to-day operational decision-making.

4.2.2.2 Developmental Recommendations for Strategic Thinkers • When making decisions you may prefer loosely defined situations and goals as they may

allow for a greater degree of freedom in decision-making. The next time you are faced with a decision, document the known information that will impact your decision as well as the related unknowns. Evaluate these unknowns and identify areas that need better definition. Ask others to help you, so that you don’t move too quickly towards a decision without needed information.


55

• Identify team members who are more detail-oriented and tactical than you are. Include these people in your decision-making so that they can help you better understand the cost and implementation issues involved in the decision.

• When considering big-picture strategies, ask a team member to play “devil’s advocate”. Rely on that individual to challenge your decisions. Ask this person to question your approach when he/she feels critical considerations are being glossed over. Doing this may help you avoid ineffective decisions that will eventually have to be abandoned. In addition, this person’s input may improve your decision-making and positively impact the immediate situation and future decisions you will make.

• After articulating a strategy, work backward to define small tactical wins that may contribute to the overall objective. Posting small wins may help others buy into your strategy.

• Consider adding more structure to your decision-making process. You only need a few replicable techniques that you can use in any decision situation (e.g., setting intermediate goals, considering main and side effects in a systematic manner, introducing regular check points as you purse overall objectives). Increasing the predictability of your decision-making process will allow others to assist you in making effective decisions and mitigate the risk their being surprised or confused by your decisions.

4.2.3 Data-Driven vs. Intuit ive Decisions Data-driven individuals tend to carefully review the relevant data and facts before making decisions. These individuals may also revisit and revise their decisions periodically to realign them with new data. Conversely, intuitive people rely more on their experience and gut instincts, preferring to make quick decisions and move on. Data-driven decision makers prefer to review large amounts of information and use it to make sound decisions. They seem to approach decisions as puzzles to be solved. They typically gather information from sources immediately related to the situation as well as those further from the decision at hand; as a result, they may become overwhelmed by the data and struggle to prioritize the facts in terms of their importance to the decision. After gathering all available information, they analyze it from many different angles, incorporating multiple perspectives and considering the individual pieces of data and the sum of the information. They value well-supported decisions over timely decisions. They may let the results of their analysis drive their decisions, even if they defy intuition, past experience, or conventional wisdom. As such, they may be most comfortable and effective in situations where there is a finite amount of information related to the decision, and those facts provide a strong foundation for the ultimate decision. However, they may struggle when there is insufficient information or when the facts do not point to the best answer. In contrast, intuitive decision-makers prefer to base their decisions on past experience and gut instincts. They may be comfortable acting without thoroughly analyzing the factors at play in each situation. They approach decision-making with confidence that their innate talent, previous experience, common sense, and business savvy will lead them to effective decisions. They tend to be comfortable choosing among possible approaches even if they are not well-informed about all possible approaches. They often take a fluid approach to


56

decisions, feeling that meticulous analysis may be unnecessary at best and detrimental at worst. They may make creative decisions because they don’t need evidence to support decision alternatives. Intuitive decision-makers may be bored by reviewing seemingly trivial information and prefer to quickly identify the decision to be made, then select and implement an approach. Such people may be admired for their confidence, but may also be seen as “flying by the seat of their pants”. They may perform well in situations where timely decisions are needed and clear evidence or supporting information is unavailable. However, at other times they may fail to consider important data and facts in their decision-making process. 4.2.3.1 Developmental Recommendations for Data-Driven Decisions • When analyzing data needed to make a decision, consider how various organizational

perspectives may affect your interpretation of the data. • Think of the last time you made a bad decision despite having thoroughly analyzed the

available evidence. Try to identify warning signs that may have preceded the decision. Did others on your team, whom you could include in your future decisions, doubt the decision?

• Consider a time when you were overwhelmed by the information you had gathered in order to make a decision. How might you have better prioritized the information? Identify techniques you can use to better manage decision-related data in future situations, such as categorizing evidence in terms of shared content, rating each piece of data as first, second, or third tier in terms of value and importance, and discarding information that may be only superficially related to the decision.

• When proposing solutions to complex problems, consider your audience. Although you may value data and analytics, others may prefer an executive summary. Highlight your critical findings and provide others with “key take-aways” related to your decision; this may be more persuasive than providing a detailed description of all of the factors contributing to your decision.

• When approaching a new decision point, set a deadline for the decision and timelines for each activity needed for the final decision. Creating a decision-making schedule may mitigate the risk of delaying critical decisions.

4.2.3.2 Developmental Recommendations for Intuitive Decisions • Before making a decision, document your reasoning in case you are asked to provide

more information on your approach. Delegate fact gathering and data analysis to team members who may be motivated by opportunities to engage in these investigative activities.

• Listen carefully when others present data that conflict with your past experience or intuition. You will show that you value their input and expertise, and heeding others’ warnings may prevent you from becoming overly confident based on hunches that paid off in the past. Considering others’ concerns about your decisions may help you challenge your own thinking, which may lead to higher-quality decisions.

• Periodically re-evaluate your decisions, noting what worked and what you might have done differently had you known then what you know now. Look specifically for decisions


57

where you missed information that might have prevented you from going down an unproductive path and having to change course afterwards.

• Consider the impact your decisions may have on your team and the wider organization. Identify the kinds of decisions you are required to make for which you may need to rely on more than your judgment and experience alone. Develop techniques to gather needed information and evaluate critical facts for those situations.

• When making a decision, think about previous experiences that may influence your approach to the situation. Reflect on the elements that were successful as well as those that could have been improved. This may mitigate the risk of repeating mistakes and/or missing the same critical considerations in multiple situations.

4.3 Decision-Making Styles People differ in their tendencies to (a) avoid threats or seek rewards, (b) think tactically or strategically, and (c) use data or intuition when making decisions. Each person’s decision-making style reflects a combination of these tendencies. For example, one person may think strategically and use data to drive decisions to avoid threat. A different person may seek rewards by focusing on tactical details and following his/her intuitions. The combinations of tendencies across these scales define eight decision-making styles. These styles provide a more holistic account of each person’s typical decision-making. Figure 4.2 presents the eight decision-making styles, which we describe below. Each decision-making style is defined by an archetypal occupation that exemplifies the decision-making characterized by the scores on the three scale pairs. These archetypal occupations are only intended to describe typical thinking patterns in decision-making, not likely vocational interests, preferences, or performance. Figure 4.2 Decision-Making Styles

4.3.1 Auditors Some individuals are characterized as threat avoidant, tactical thinking, and data-driven decision-makers. This pattern of decision-making is best represented by “Auditors” because such people typically make tactical decisions designed to minimize short-term risks, pay


58

attention to details, get involved with day-to-day operations, and make data-based decisions that they periodically review and adjust. 4.3.2 Surgeons Individuals characterized as threat avoidant, tactical thinking, and intuitive decision-makers are labeled as “Surgeons”. They typically avoid taking unnecessary risks, focus on solving immediate problems, pay attention to details, base their decisions on practical experience, and prefer hands-on involvement in the work. 4.3.3 Stock Traders Some individuals seek tactical, short term rewards using data-driven decisions. “Stock Traders” are prototypical of such decision-making tendencies because they focus on short-term wins, making quick and calculated tactical decisions based in data, and regularly evaluating the effectiveness of their decision-making processes. 4.3.4 Defense Analysts “Defense Analysts” make data driven decisions to defend comprehensively against a broad range of long term threats. They are focused on strategies that systematically eliminate threats; they base their decisions on a thorough review of the data, and periodically evaluate their decision-making methods. 4.3.5 Polit icians Some individuals make quick and intuitive decisions, guided by a long term strategy, designed to maximize their pay-offs. This pattern of decision-making is best represented by “Politicians” because these individuals tend to remain alert for opportunities to advance their strategic agendas, make decisions to maximize long-term interests, base decisions on instinct, and generally ignore details such as implementation issues. 4.3.6 Chess Players “Chess Players” are people who avoid long term strategic threats by making rapid decisions based on experience and intuition. We chose this prototype because such people make decisions designed to minimize threats to future positions, base decisions on strategy, think several moves ahead, make decisions based on their past experience and intuition, accept short-term losses to make long-term gains, and expect that success will take time. 4.3.7 Promoters Some individuals are characterized as reward seeking, tactical thinking, and intuitive decision-makers. This pattern of decision-making is best represented by “Promoters” because such people tend to make rapid decisions designed to maximize immediate rewards, expect quick results, make decisions based on their experience and intuition,


59

spend little time worrying about past mistakes, and show little interest in detailed data analyses. 4.3.8 Investors Finally, “Investors” are people who seek sustainable, long term rewards by making careful, data-based decisions. We chose this prototype because such people focus on achieving long-term strategic goals, stay alert for hidden but consistent business trends, accept short-term losses to earn long-term gains, make data-driven decisions, periodically revisit their decisions, and try to maximize their potential rewards. 4.4 Reactions to Feedback Because the base rate for bad business decisions is at least 50% (DeVries, 1992), a crucial element of business judgment concerns how people react to feedback that their decisions didn’t work out. People tend to respond to negative feedback in one of three typical ways: (a) by being defensive or remaining cool-headed; (b) by denying or accepting the feedback; and (c) with superficial agreement or genuine engagement in the feedback. Every individual can be described in terms of these three dimensions. More importantly, unlike the pre-decision approaches which have unique strengths at both the low and high ends of each scale, reaction to feedback scales include undesirable behaviors at the low end (i.e., defensiveness, denial, superficial agreement) and desirable behaviors at the high end (i.e., remaining cool-headed, accepting feedback, genuinely engaging in feedback). The Hogan Judgment Report describes the degree to which people typically rely on these three tendencies when responding to feedback about failed decisions. Understanding these tendencies can enhance an individual’s strategic self-awareness. This information has implications for the selection, placement, and development of organizational leaders. Knowing how an individual typically reacts to negative feedback also provides a realistic preview of their responses to developmental initiatives such as coaching. 4.4.1 Defensive vs. Cool-Headed Defensive individuals tend to respond to negative feedback by becoming emotional and projecting blame outwards to people, circumstances, timing, and other factors outside their control. Such individuals may need time to compose themselves before responding to negative feedback, and need to find ways to suppress their inclination to react emotionally. Conversely, cool-headed individuals tend to remain calm and consider how they may have contributed to incorrect business decisions. Although this response is preferable to defensiveness, cool-headed individuals may sometimes benefit from demonstrating more passion when defending their decisions, especially when the facts are on their side. They might also consider displaying more concern about the negative outcomes to which they contributed. Otherwise, they may seem unconcerned about the news that their decision did not work out, and their long-term commitment to their job may be questioned based on that perceived lack of passion.


60

4.4.2 Denial vs. Acceptance People who deny negative feedback tend to refuse to recognize facts, deflect responsibility, reinterpret failure as success, and deny that problems exist, or try to move on. They could benefit from facing criticism and attending to feedback in the future. They could also reflect on how others may interpret their denial as an inability to learn from experience. They can use such events as opportunities to learn and improve future decisions. Conversely, individuals who can accept negative feedback tend to carefully consider the facts, address failure directly, and leverage negative feedback to improve future decisions. Although this response is preferable to denying negative feedback, accepting individuals may also benefit from using feedback to learn about what they did right, not just where they made mistakes. In addition, these individuals may be too accepting of negative feedback at times. As such, they should remember that taking responsibility for factors beyond their control is just as counterproductive as not taking responsibility for factors within their control. Accepting negative feedback too quickly can also send the wrong message to others. 4.4.3 Superficial vs. Genuine Engagement People who are superficially engaged seem open to negative feedback, willing to admit failure, and solicit advice for future improvement. In reality, however, they are purposefully attempting to gain acceptance and avoid further confrontation about the problem. Superficially engaged individuals may benefit from broadening their base of feedback and develop a list of action items they are willing to take on to improve. More importantly, they should make sure to have an accountability structure set up for individual improvement. Conversely, genuinely engaged individuals tend to actively participate in feedback sessions about bad decisions and commit themselves to finding ways to improve future decisions. Although this response is preferable to superficial engagement, genuinely engaged individuals may also benefit from thinking about times when their desire to improve may have resulted in their acting on feedback that may not have been constructive or even valid. They may consider what they can do to evaluate the quality of post-decision feedback they receive before acting on it. 4.5 Feedback Response Patterns As previously noted, people vary in their reactions to feedback by (a) becoming defensive or remaining cool-headed, (b) denying or accepting criticism, and (c) superficially or genuinely engaging with the feedback. However, these reactions do not exist in isolation. Instead, each person exhibits a combination of reactions that define a response pattern. For example, one person may react defensively, deny problems, and superficially engage in feedback, whereas a different individual may remain cool-headed, accept criticism, and genuinely engage in feedback. The possible combinations of response tendencies based on these scales allows us to define eight feedback response styles. These styles provide a more holistic account of each person’s typical reactions to negative feedback. However, unlike our eight decision-making styles, these eight feedback response patterns are not described by archetype occupations; rather, we describe these eight feedback response


61

patterns in the simplest terms, using the scores comprising each pattern. We describe each pattern below. 4.5.1 Defensive, Denial, Superficial Some individuals respond to negative feedback by becoming defensive, denying or deflecting criticism, and superficially engaging in feedback sessions. Such individuals tend to blame undesirable outcomes on other people or external factors, react emotionally to bad news, and avoid taking responsibility for unwelcome results. They may become annoyed with negative feedback and/or quickly forget it, inadvertently train others not to deliver bad news, have difficulty learning from experience, and resist coaching and other development efforts. 4.5.2 Defensive, Acceptance, Superficial Some people tend to respond to negative feedback by becoming defensive at first, then accepting the criticism, but only superficially engaging in feedback. They are initially upset by negative feedback, seeing it as a form of rejection. They defend themselves by blaming other people and external circumstances for the mistakes. However, after the initial resistance has passed, they may see the negative feedback as legitimate. Nonetheless, due to their superficial engagement, they often avoid taking complete responsibility for undesirable outcomes, pretend to agree with feedback to get along, and have difficulties changing their behavior following feedback. 4.5.3 Defensive, Denial, Genuine Some individuals respond to negative feedback by becoming defensive, denying or deflecting criticism, but ultimately engaging in feedback. Such people tend to initially react emotionally, blame other people, deflect or deny any mistakes, and have difficulties accepting criticism. Ironically, in the context of denying any problems, they will actively participate in feedback sessions and may benefit from coaching after the initial emotional reaction has subsided. 4.5.4 Defensive, Acceptance, Genuine Other people tend to respond to negative feedback by becoming defensive at first, then accepting the criticism and genuinely engaging in feedback sessions. Such individuals tend to first react defensively and emotionally to negative feedback and blame other people or external factors for the mistakes. However, once the initial emotional response has passed, they may be more modest and straightforward in considering feedback, willing to reflect on past mistakes, remain genuinely engaged in feedback sessions, appear motivated to improve performance and change behavior, and benefit from coaching. 4.5.5 Cool-Headed, Denial, Superficial Some individuals respond calmly to negative feedback, but deny criticism and superficially engage in feedback. Such individuals tend to receive bad news calmly, are confident in their


62

decision-making, lack a sense of urgency to correct past mistakes, disregard negative feedback, prefer not to dwell on past mistakes, show skill in reinterpreting failure as success, have difficulties learning from experience, and avoid taking personal responsibility for failed decisions. 4.5.6 Cool-Headed, Denial, Genuine Other people tend to respond to negative feedback by remaining calm and genuinely engaging in feedback, but denying criticism. Such individuals tend to remain composed in the face of bad news, rarely take bad news personally, and deflect or deny feedback with which they disagree. However, they will genuinely engage in feedback sessions and may benefit from coaching once a trusting environment is formed that encourages them to safely acknowledge their mistakes. 4.5.7 Cool-Headed, Acceptance, Superficial Some individuals respond to negative feedback by remaining calm and accepting criticism, but only superficially engaging in feedback. Such people tend to take bad news well, avoid taking negative feedback personally, accept past failures, avoid dwelling on the past, and appear willing to engage in feedback sessions. They will seem to listen to and agree with feedback, but may have difficulties actually changing future behavior based on this coaching. 4.5.8 Cool-Headed, Acceptance, Genuine Finally, other people tend to respond to negative feedback by remaining cool-headed, accepting criticism, and genuinely engaging in feedback sessions. Such individuals tend to remain composed in the face of bad news, accept negative feedback in an open-minded and non-defensive manner, accept personal responsibility for past mistakes, directly address previous failures to improve future decisions, genuinely engage in feedback sessions, and carefully reflect on feedback. As this description suggests, these individuals can most easily benefit from coaching because their natural response tendencies do not create internal obstacles. 4.6 Openness to Feedback and Coaching As noted above, people demonstrate a variety of reactions to negative feedback, many of which are counterproductive. With combinations of the pre-decision approach scales, recall that our interest was in using those eight combinations to describe distinct decision-making styles. However, with the response to feedback scales, we are less interested in defining eight response styles and more interested in describing how each individual’s combination of scores across these scales define his or her overall openness to feedback and coaching. By combining people’s scores across the three feedback scales, we can describe their overall openness to feedback and coaching to improve future decisions. For example, one person may generally resist coaching, whereas another individual may be generally receptive to these efforts. This is crucial information because it concerns the likelihood that a person will be able to learn from his or her mistakes to improve future business decisions. Put


63

another way, this information concerns a person’s coachability, a crucial element of both good judgment and effective performance. In general, individuals can be characterized as resistant, neutral, or receptive to feedback and coaching. Although being receptive is generally preferable, each group has its own set of strengths and challenges. We describe each below. 4.6.1 Resistant Feedback resistant individuals may be well-suited for making difficult decisions and standing by them despite criticism. However, unmitigated resistance to feedback and coaching over time may prevent others from sharing candid feedback, erode team dynamics, and impede effective performance. Individuals characterized as feedback resistant are not “un-coachable”, but must consistently monitor their counterproductive reactions to ensure that they mitigate them to change their behavior and improve future decisions. As such, in response to feedback, resistant individuals should focus on not taking criticism personally or reflexively denying or deflecting feedback. Instead, they should take time to carefully consider the feedback before responding. If they can think more deeply about their role in these mistakes, they may be able to find solutions that will help them make better decisions in the future. To facilitate these conditions, it is important to establish a safe and trusting feedback environment. For example, coaches may need to spend more time up front establishing credibility when working with resistant individuals. It is also important to set realistic expectations and provide positive reinforcement when these individuals act on feedback. 4.6.2 Neutral Feedback neutral individuals are often good at recognizing instances where they may be responsible for bad decisions. However, they may alternate between calm and emotional reactions, acceptance and denial of criticism, and active and inactive engagement in feedback sessions. In short, these individuals demonstrate a middle of the road response to feedback, neither resisting it nor accepting responsibility entirely. In response to negative feedback, neutral individuals should carefully consider times when they may have avoided or deflected feedback before giving it the full consideration it deserved. They should also take time to explore the feedback and their role in past mistakes before responding. In addition, they may benefit from reflecting on times when they could have more actively participated in feedback sessions. Insights gained from these efforts may potentially help such individuals take full advantage of future opportunities to improve decision-making. 4.6.3 Receptive Feedback receptive individuals seem willing to accept fault for their failed decisions and benefit from coaching. However, their tendency to internalize fault may also cause them to take responsibility for others’ errors as well as external factors beyond their control. As such,


64

these individuals should carefully monitor their reactions to feedback to ensure that they are not internalizing excess responsibility for undesirable outcomes. Feedback receptive individuals should also consider what roles they and others have played in past failed decisions. These reflections should help them focus on feedback about their performance without taking blame for others’ mistakes. Also, instead of automatically internalizing all feedback, they should consider feedback on a case-by-case basis to determine its merit. Through these efforts, feedback receptive individuals can leverage constructive disagreement and debate to further explore solutions to previous mistakes and improve future business decisions.

5. ADMINISTERING THE HOGAN JUDGMENT REPORT

65

5. Administering the Hogan Judgment Report Hogan Assessment Systems, Inc. (“Hogan”) provides a state-of-the-art administration platform developed to meet the needs of our clients. Hogan designed this platform to maximize security, ease of use, speed, and flexibility. This allows us to accommodate clients of all sizes. An overview of key features of this system is presented below. For further information, please contact Hogan’s Customer Service Department at 1-800-756-0632 or [email protected]. Office hours are 8am-5pm U.S. central time and after-hours messages are checked daily. 5.1 Key Features of the Online Assessment It is important that test administrators: (a) understand how participants complete an assessment, (b) answer questions or concerns participants may raise, and (c) use test administrator tools effectively. To address these issues, Hogan trains administrators in the functionality of the online assessment system. In the initial training session, we instruct new administrators on how to create participant IDs and use other tools on the administrative website. Additional training is available for creating participant groups, obtaining reports, changing report options, and specifying report delivery options. The Hogan testing system is fully redundant, using multi-location systems architecture to ensure constant availability. Clients can access the testing platform 24 hours a day, 7 days a week from any internet-enabled device. Test results are normally delivered in 90 seconds or less, making results nearly instantaneous. The system delivers results via the web or encrypted PDF files sent by e-mail. Hogan consults with outside security experts to ensure data security. We use 2,048-bit secure access via password protection when safeguarding clients’ and users’ assessment data. All Hogan web ordering systems allow us to tailor the ordering and reporting experience to each user based on a hierarchical system of client and user preferences. Whether a client orders from a single office or locations across the globe, all orders can flow through a single account. Hogan’s product-level security features allow clients to restrict individual users’ ability to order and view reports on a product-by-product basis. 5.2 Completing the Assessment Using the Online Platform Although test taking time may vary depending on a user’s reading speed, the Hogan Judgment Assessment usually requires about 20 minutes to complete. Once a participant receives a Hogan ID from the administrator, he/she logs onto the website at http://www.gotohogan.com or a customized portal designed specifically for a Hogan client. To log onto the website, a minimum version of Microsoft Internet Explorer 7, Chrome, Safari, Firefox or other browser that supports cookies and JavaScript is needed. Once on the website, the individual sees a login page similar to the one presented in Figure 5.1.


66

Figure 5.1 Participant Login Web Page

On the login page, the participant is asked to enter his/her assigned Hogan ID and password (e.g., BB123456 and SAMPLE) and click the “Go” button. The participant is then prompted to complete a brief set of demographics questions (see Figure 5.2) and agree to an informed consent clause (see section 5.3). This clause outlines information regarding the purpose, administration, and results of the assessment.


67

Figure 5.2 Participant Information Web Page

Once users have logged in, the system asks them to create a personal password and complete additional information fields. When all fields are complete, they must click “Go” and are directed to the participant menu page. This page displays each assessment they must complete (see Figure 5.3). After users complete an assessment, the system returns them to this page if they need to select and complete additional assessments.


68

Figure 5.3 Participant Menu Wed Page

5.2.1 Numerical Section The numerical section of the Hogan Judgment Assessment consists of 15 multiple-choice numerical problems. Test takers have 10 minutes to complete as many of these problems as they can. Participants can use scratch paper and a pencil or pen when completing this section, but cannot use calculators or other external devices. Figure 5.4 presents an example of instructions from the numerical section.


69

Figure 5.4 Numerical Section Instructions

Following the instructions, participants receive two sample questions to familiarize them with the format of the items. For each sample item, the system also indicates the correct response. Figure 5.5 presents an example of a numerical sample item. Once participants complete the sample questions, they can select “Next” to start the numerical section or “Stop Assessment” to quit. After they begin this section, they cannot go back and finish it at a later time.


70

Figure 5.5 Numerical Section Sample Item

Once participants start the numerical section, they can click “Next” or “Back” to navigate between questions. They can also skip questions and go back to review previous questions before their time runs out. An on-screen timer displays the time remaining and flashes red to warn participants when they are nearly out of time. Figure 5.6 presents an example of a page from the numerical section.


71

Figure 5.6 Numerical Section Sample Page

If participants complete all questions with time remaining, they will see a question review page. From this page, they can revisit skipped questions or review questions they previously answered. The review page presents buttons for each of the 15 items, with unanswered questions appearing in red. If a participant selects a number, the system redirects them to that question. Answering that question or selecting “Next” returns the participant to the review page. Figure 5.7 presents an example of the review page.


72

Figure 5.7 Numerical Section Review Page

If the 10-minute timer runs out before participants answer all questions, the system automatically saves their answers and provides a message instructing them that their time has expired and their responses have been recorded (see Figure 5.8). Participants click “Next” on this message to proceed to the next section of the Hogan Judgment Assessment.


73

Figure 5.8 Out of Time Page

5.2.2 Verbal Section The verbal section of the Hogan Judgment Assessment contains 48 statements pertaining to letters in the alphabet, analogies, types of items, opposites, sizes, and meanings, and asks participants to indicate if the statements are true or false. Participants have two minutes to respond to as many statements as they can. Figure 5.9 presents an example of instructions from the verbal section.


74

Figure 5.9 Verbal Section Instructions

Following the instructions, participants view two sample questions to familiarize them with the format of the items. For each sample item, the system also indicates the correct response. Figure 5.10 presents an example of a verbal sample item. Once participants complete the sample questions, they can select “Next” to start the verbal section or “Stop Assessment” to quit. After they begin this section, they cannot go back and finish it at a later time.


75

Figure 5.10 Verbal Section Sample Item

Once participants start the verbal section, they can click “Next” or “Back” to navigate between questions. They can also skip questions and go back to review previous questions before their time runs out. An on-screen timer displays the time remaining and flashes red to warn participants when they are nearly out of time. Figure 5.11 presents an example of a page from the verbal section.


76

Figure 5.11 Verbal Section Sample Page

If participants complete all questions with time remaining, they will see a question review page. From this page, they can revisit skipped questions or review questions they previously answered. The review page presents buttons for each of the 48 items, with unanswered questions appearing in red. If a participant selects a number, the system redirects them to that question. Answering that question or selecting “Next” returns the participant to the review page. Figure 5.12 presents an example of the review page.


77

Figure 5.12 Verbal Section Review Page

If the two-minute timer runs out before participants answer all questions, the system automatically saves their answers and provides a message instructing them that their time has expired and their responses have been recorded (see Figure 5.8). Participants click “Submit” on this message to proceed to the next section of the Hogan Judgment Assessment.


78

5.2.3 Decision-Making Style Section The decision-making style section of the Hogan Judgment Assessment contains 75 true/false statements. This section is untimed and has no correct or incorrect responses. Instead, participants read each statement and respond “True” if that statement accurately describes them or “False” if it does not. Although participants are not required to answer every question, scores cannot be computed and results will not be presented if the participant skips more than one third of the items on any scale. Figure 5.13 presents an example of instructions from the decision-making style section. Figure 5.13 Decision-Making Style Section Instructions

Following the instructions, participants receive two sample questions to familiarize them with the format of the items. For each sample item, the system also provides feedback to reiterate the fact that this section has no correct or incorrect responses and responses reflect general preferences. Figure 5.14 presents an example of a sample item from the decision-making style section.


79

Figure 5.14 Decision-Making Style Section Sample Item

Once participants start the decision-making style section, they can click “Next” or “Back” to navigate between questions. They can also skip questions and go back to review previous questions. Figure 5.15 presents an example of a page from the decision-making style section.


80


81

Figure 5.15 Decision-Making Style Section Sample Page

Once participants complete all questions, they will see a question review page. From this page, they can revisit skipped questions or review questions they previously answered. The review page presents buttons for each of the 75 items, with unanswered questions appearing in red. If a participant selects a number, the system redirects them to that question. Answering that question or selecting “Next” returns the participant to the review page. Figure 5.16 presents an example of the review page.


82

Figure 5.16 Decision-Making Style Section Review Page

Participants can start and stop this section of the assessment at their convenience. If they stop the assessment, the system retains all information submitted to that point. They can log back into the system with their user ID and self-created personal password to continue the assessment at any time. Once they are satisfied with their responses, they can submit them to complete the last section of the Hogan Judgment Assessment. A scoring engine automatically processes the participant’s responses and sends reports to one or more e-mail addresses as designated by the administrator. If the account administrator or participant experiences a problem, they can contact Hogan’s Customer Service Department at 1-800-756-0632 or [email protected]. 5.3 Participant Informed Consent Hogan requires that all individuals taking its assessments give their informed consent to participate in the assessment process. This is the fundamental concept that underlies all current and anticipated data protection protocols and legislation. In order for individuals taking the assessments to give their informed consent, they must understand the purpose of the assessment, the likely use of the assessment data, and how the data are protected. These protocols are described below and are binding on all Hogan clients and individuals taking the assessments. Failure to comply with any of these safeguards will constitute grounds for termination of any agreements in place between Hogan and the person(s) or entity(ies) concerned.


83

5.3.1 Consent By initiating the assessment(s), the participant expressly consents to the processing, including storage, copying, deletion, printing, transfer, aggregation and use, of the data he/she provides as set forth below. This consent is given voluntarily and the participant may withdraw such consent at any time. Hogan shall not be responsible for or subject to any liability resulting from any change to the participant’s potential or continued employment status that may result from either (a) their taking of the assessment(s), or (b) their decision to withdraw their consent to the processing of the data he/she provides. 5.3.2 Purpose Hogan’s assessments were created to provide personal characteristic information and feedback to trained and accredited consultants and HR professionals. The data participants provide, including the replies they submit during the assessment(s), will be processed by Hogan. The results of the assessment(s) are primarily used for selection and/or development by the requesting organization. 5.3.3 Data Processing Hogan has the right to process the data participants provide, including the storage, copying, deletion, printing, transfer, aggregation and use thereof. Hogan may also transfer, via secured methods, the data to trained and accredited consultants or HR professionals for selection and/or development purposes. In addition, Hogan may use anonymously held (i.e., identifying information removed) aggregated data for its own research purposes. All Hogan clients are responsible for complying with national and international protocols covering data use and storage. For additional information on Hogan’s data privacy policy and its compliance with the U.S.-EU Safe Harbor and the U.S.-Swiss Safe Harbor Frameworks, please visit www.hoganassessments.com/privacy-policy for more information. 5.3.4 Access to Data The dissemination of assessment results is the sole responsibility of the requesting organization. Hogan will only provide results to individuals taking the assessments under specific direction from the requesting organization. Individuals taking the assessments are not guaranteed access to their individual results. However, they may withdraw their consent to the processing of the data they provided at any time, and such data will be deleted from Hogan’s systems. If participants choose to withdraw their consent to the processing of the data they provide after Hogan has provided their assessment results to the requesting organization, Hogan shall have no obligation to ensure the deletion of any data that may be contained in such assessment results. In addition, Hogan shall not be responsible for or subject to any liability resulting from any change to the participant’s potential or continued employment status that may result from their decision to withdraw their consent.


84

5.3.5 Security In order to safeguard assessment results, the website contains only the assessment items, not the assessment programs (which are held by Hogan and its clients). It is impossible to process assessment results through the website. Assessment results can only be processed by downloading the raw data, decrypting it, and scoring these data with appropriate programs. Until that time, responses to assessment items are merely encrypted alphanumeric strings with no discernible meaning. 5.3.6 Contact If participants have any questions about informed consent, the purpose of the assessment(s), the processing of the data they provide, or Hogan’s privacy policy, they can contact [email protected] or their local Hogan distributor. 5.4 Accommodating Individuals with Disabil it ies The Americans with Disabilities Act of 1990 (ADA) prohibits employment discrimination against qualified individuals with disabilities. This law has important implications for employers’ procedures used in interviewing, testing, and hiring new employees. For pre-employment testing, the ADA specifies that employers must provide alternate forms of employment testing that “accurately (assess) the skills, aptitudes, or whatever other factor of such applicant or employee that such test purports to measure, rather than reflecting the impaired sensory, manual or speaking skills of such employee or applicant” Sec. 102(b)(7), 42 U.S.C.A.Sec. 12112. Hogan complies with the ADA requirements by working with clients to accommodate individuals with special needs. Large print assessments are available from Hogan’s Customer Service Department at 1-800-756-0632 or by e-mail at [email protected]. Hogan can make additional accommodations on a case-by-case basis. 5.5 Frequently Asked Questions The following section presents frequently asked questions and answers to those questions. Q. I am trying to sign back in to complete the assessment but my user ID and password are not working. A. On the initial participant information screen, you were asked to change the initial password you were given to a personal password you could more easily remember. Please use the new personal password you created to log back into the system. Q. Is it a timed assessment? A. The numerical and verbal sections of the Hogan Judgment Assessment have ten- and two-minute timers, respectively. The decision-making style section is untimed and you can take as much time as needed to complete this section.


85

Q. Can I stop the assessment at any time? A. Once you begin the numerical and verbal sections of the assessment, you must complete them. However, during the decision-making style section you may stop at any time and return at your convenience. You will need your user ID and new personal password to log back into the website. Q. How long will the assessment take? A. We anticipate that it should take you about 20 minutes to complete the Hogan Judgment Assessment. Q. Will I receive a copy of my results? A. We are not at liberty to share or discuss results with participants. Results are sent to the company that asked you to complete the assessment. That company decides whether or not to share results with you. Q. Will all my data be lost if my system freezes before I complete the assessment? A. No, your responses are saved after each page is completed.

REFERENCES

86

REFERENCES Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence

for overlapping traits. Psychological Bulletin, 121, 219. Aguinis, H., Henle, C. A., & Ostroff, C. (2001). Measurement in work and organizational

psychology. Handbook of industrial, work and organizational psychology, 1, 27-50. Aitkenhead, M. (1984). Impression-management and consistency effects in the processing

of feedback. British Journal of Social Psychology, 23, 213-222. doi:10.1111/j.2044-8309.1984.tb00632.x

American Educational Research Association, American Psychological Association, & the

National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.

American Psychiatric Association (APA). (2013). Diagnostic and statistical manual of mental

disorders (5th ed.). Washington, DC: Author. Americans with Disabilities Act of 1990, 102(b)(7), 42 U.S.C.A. 12112. Ariely, D. (2008). Predictably irrational: The hidden forces that shape our decisions. New

York, NY: HarperCollins. Aronson, E., Wilson, T. D., & Akert, R. M. (2010). Social Psychology. Upper Saddle River, NJ:

Prentice Hall. Ashford, S. J., & Northcraft, G. B. (1992). Conveying more (or less) than we realize: The role

of impression-management in feedback-seeking. Organizational Behavior and Human Decision Processes, 53, 310-334.

Atkinson, J. W. (1957). Motivational determinants of risk-taking behavior. Psychological

Review, 64, 359-372. doi:10.1037/h0043445 Axelrod, A. (2006). Profiles in audacity: Great decisions and how they were made. New

York, NY: Sterling Publishing Company, Inc. Baddeley, A. D. (1986). Working memory. Oxford: Clarendon. Baddeley, A. D. (1997). Human memory: Theory and practice. Psychology Press. Barrick, M. R., Stewart, G. L., & Piotrowski, M. (2002). Personality and job performance: Test

of the mediating effects of motivation among sales representatives. Journal of Applied Psychology, 87, 43–51.

REFERENCES

87

Baumeister, R., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5, 323–370.

Bechara, A. (2005). Decision-making, impulse control and loss of willpower to resist drugs: a

neurocognitive perspective. Nature Neuroscience, 8, 1458-1463. Bell, S. T., & Arthur, W. R. (2008). Feedback acceptance in developmental assessment

centers: The role of feedback message, participant personality, and affective response to the feedback session. Journal of Organizational Behavior, 29, 681-703. doi:10.1002/job.525

Benartzi, S., & Thaler, R. H. (1995). Myopic loss aversion and the equity premium puzzle.

Quarterly Journal of Economics, 110, 73. Benet-Martinez, V., & John, O. (1998). Los Cinco Grandes across cultures and ethnic groups:

Multitrait-multimethod analyses of the Big Five in Spanish and English. Journal of Personality and Social Psychology, 75, 729-750.

Berry, C. M., Ones, D. S., & Sackett, P. R. (2007). Interpersonal deviance, organizational

deviance, and their common correlates: a review and meta-analysis. Journal of Applied Psychology, 92, 410.

Betsch, C. (2008). Chronic preferences for intuition and deliberation in decision-making:

Lessons learned about intuition from an individual differences approach. In H. Plessner, C. Betsch, T. Betsch (Eds.), Intuition in Judgment and Decision-making (pp. 231-248). Mahwah, NJ US: Lawrence Erlbaum Associates Publishers.

Borman, W. C., Penner, L. A., Allen, T. D., & Motowidlo, S. J. (2001). Personality predictors of

citizenship performance. International Journal of Selection and Assessment, 9, 52-69.

Brett, J. F., Atwater, L. A. (2001). 360-degree feedback: Accuracy, reactions and perceptions

of usefulness. Journal of Applied Psychology, 86, 930–942. British Psychological Society (2007). Hogan Personality Inventory. British Psychological

Society Psychological Testing Centre test reviews. London, UK: Author. Brocas, I. (2012). Information processing and decision-making: Evidence from the brain

sciences and implications for economics. Journal of Economic Behavior and Organization, 83, 292-310. doi:10.1016/j.jebo.2012.06.004

Bruine de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult

decision-making competence. Journal of Personality and Social Psychology, 92, 938. Bruine de Bruin, W., Del Missier, F., & Levin, I. P. (2012). Individual differences in decision-

making competence. Journal of Behavioral Decision-making, 25, 329-330.

REFERENCES

88

Busenitz, L. W., & Barney, J. B. (1997). Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of Business Venturing, 12, 9-30.

Button, S., Mathieu, J., & Zajac, D. (1996). Goal orientation in organizational research: A

conceptual and empirical foundation. Organizational Behavior and Human Decision Processes, 67, 26-48.

Campbell, D. P., Hyne, S. A., & Nilsen, D. L. (1992). Campbell Interest and Skill Survey

manual. Minneapolis, MN: National Computer Systems, Inc. Carver, C. S., & White, T. L. (1994). Behavioral inhibition, behavioral activation, and affective

responses to impending reward and punishment: the BIS/BAS scales. Journal of Personality and Social Psychology, 67, 319.

Chamorro-Premuzic, T. (2008). The numerical reasoning test 20-items (NRT-20). London:

Goldsmiths, University of London (unpublished test). Chen, G., Crossland, C., & Luo, S. (2014). Making the same mistake all over again: CEO

overconfidence and corporate resistance to corrective feedback. Strategic Management Journal. Forthcoming.

Choudhry, N. K., Fletcher, R. H., & Soumerai, S. B. (2005). Systematic review: The

relationship between clinical experience and quality of health care. Annals of Internal Medicine, 142, 260–273.

Citroen, C. L. (2011). The role of information in strategic decision-making. International

Journal of Information Management, 31, 493-501. doi:10.1016/j.ijinfomgt.2011.02.005

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in scale development.

Psychological Assessment, 7, 309-319. Cloninger, C., Przybeck, T., Svrakic, D., & Wetzel, R. (1994). TCI—the temperament and

character inventory: A guide to its development and use. St Louis (MO): Center for Psychobiology of Personality, Washington University.

Conn, S. R., & Rieke, M. L. (1994). The 16PF fifth edition technical manual. Champaign, IL:

Institute for Personality and Ability Testing. Cooper, A. C., Woo, C. Y., & Dunkelberg, W. C. (1988). Entrepreneurs' perceived chances for

success. Journal of Business Venturing, 3, 97-108. Costa, P. T. Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and

NEO Five-Factor Inventory (NEO-FFI): Professional manual. Odessa, FL: Psychological Assessment Resources.

REFERENCES

89

Davis, C., Patte, K., Tweed, S., & Curtis, C. (2007). Personality traits associated with decision-making deficits. Personality and Individual Differences, 42, 279–290.

De Neys, W., Novitskiy, N., Geeraerts, L., Ramautar, J., & Wagemans, J. (2011). Cognitive

control and individual differences in economic ultimatum decision-making. PloS one, 6, e27107. doi:10.1371/journal.pone.0027107

De Raad, B., & Perugini, M. (Eds.). (2002). Big Five assessment. Seattle, WA: Hogrefe &

Huber. DeVillis, R. F. (1991). Scale development: Theory and application. Newbury Park, CA: Sage. DeVries, D. L. (1992). Executive selection: Advances but no progress. Issues and

Observations, 12, 1-5. Dewberry, C., Juanchich, M., & Narendran, S. (2013). Decision-making competence in

everyday life: The roles of general cognitive styles, decision-making styles and personality. Personality and Individual Differences, doi:10.1016/j.paid.2013.06.012

Donovan, J. J., & Williams, K. J. (2003). Missing the mark: Effects of time and causal

attributions on goal revision in response to goal-performance discrepancies. Journal of Applied Psychology, 88, 379–390.

Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement.

Psychological Bulletin, 86, 335-337. Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006). A meta-analytic

investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, 40–57.

Elliot, A. J., & Thrash, T. M. (2002). Approach-avoidance motivation in personality: approach

and avoidance temperaments and goals. Journal of Personality and Social Psychology, 82, 804.

Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor,

& Department of Justice (1978). Uniform guidelines on employee selection procedures. Federal Register, 43, 38290-38315.

Feather, N. T. (1968). Change in confidence following success or failure as a predictor of

subsequent performance. Journal of Personality and Social Psychology, 9, 38. Finucane, M. L., Alhakami, A., Slovic, P., & Johnson, S. M. (2000). The affect heuristic in

judgments of risks and benefits. Journal of Behavioral Decision-making, 13, 1-17.

REFERENCES

90

Fischhoff, B. (1982). Debiasing. In D. Kahneman, P. Slavic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases. New York: Cambridge Univ. Press.

Freud, A. (1946). The ego and the mechanisms of defense. New York: International Universities Press. (Original work published 1936).

Gallo, A. (2012, May 15). How to get feedback when you're the boss [Web log message].

Retrieved from https://hbr.org/2012/05/how-to-get-feedback-when-youre Gigerenzer, G., & Gaissmaier, W. (2011). Heuristic decision-making. Annual Review of

Psychology, 62, 451-482. Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring

the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28). Tilburg, Netherlands: Tilburg University Press.

Goldberg, L. R. (2008). The Eugene-Springfield Community Sample: Information available

from the research participants. Oregon Research Institute Technical Report Vol. 48, No. 1. Eugene, Oregon: Oregon Research Institute.

Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C.R., & Gough,

H.C. (2006). The International Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84-96.

Gough, H. G. (1975). Manual for the California Psychological Inventory. Palo Alto, CA:

Consulting Psychologists Press. Gough, H. G. (1996). CPI manual. Palo Alto, CA: Consulting Psychologists Press. Gruys, M. L., & Sackett, P. R. (2003). Investigating the dimensionality of counterproductive

work behavior. International Journal of Selection and Assessment, 11, 30-42. Hamers, J. H., Ruijssenaars, A. J. J. M., & Sijtsma, K. (Eds.). (1992). Learning potential

assessment. CRC Press. Harnish, V. (2012). The greatest business decisions of all time: How Apple, Ford, IBM,

Zappos, and others made radical choices that changed the course of business. New York, NY: Fortune Books.

Hodgkinson, G. P., Maule, A. J., Bown, N. J., Pearman, A. D., & Glaister, K. W. (2002). Further

reflections on the elimination of framing bias in strategic decision-making. Strategic Management Journal, 23, 1069-1076.

Hogan Assessment Systems (2014). Hogan Development Survey form 5 technical

supplement. Tulsa, OK: Hogan Assessment Systems.

REFERENCES

91

Hogan, J., Barrett, P., & Hogan R. (2007). Personality measurement, faking, and employment selection. Journal of Applied Psychology, 92, 1270-1285.

Hogan, J., & Hogan, R. (2010). Motives, Values, Preferences Inventory manual: 2010

administrative and norming updates. Tulsa, OK: Hogan Press. Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance

relations: A socioanalytic perspective. Journal of Applied Psychology, 88, 100-112. Hogan, R. (1983). A socioanalytic theory of personality. In M. M. Page (Ed.), 1982 Nebraska

symposium on motivation (pp. 55-89). Lincoln: University of Nebraska Press. Hogan, R. (1991). Personality and personality measurement. In M. D. Dunnette & L. M.

Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 2, 2nd ed., pp. 327-396). Palo Alto, CA: Consulting Psychologists Press.

Hogan, R. (1996). A socioanalytic perspective on the five-factor model. In J. S. Wiggins (Ed.),

The five-factor model of personality (pp.163-179). New York: Guilford. Hogan, R., & Hogan, J. (1995). Hogan Personality Inventory manual (2nd ed.). Tulsa, OK:

Hogan Assessment Systems. Hogan, R., & Hogan, J. (2007). Hogan Personality Inventory manual (3rd ed.). Tulsa, OK:

Hogan Assessment Systems. Hogan, R., & Hogan, J. (2009). Hogan Development Survey manual (2nd ed.). Tulsa, OK:

Hogan Press. Hogan, R., Hogan, J., & Roberts, B. W. (1996). Personality measurement and employment

decisions: Questions and answers. American Psychologist, 51, 469-477. How employees process feedback [Web log message]. (2013, March 28). Retrieved from

http://blog.talkdesk.com/how-employees-process-feedback Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: the Big Five

revisited. Journal of Applied Psychology, 85, 869. Ilgen, D. R., Fisher, C. D., & Taylor, M. S. (1979). Consequences of individual feedback on

behavior in organizations. Journal of Applied Psychology, 64, 349. Ilies, R., & Judge, T. A. (2005). Goal regulation across time: The effects of feedback and

affect. Journal of Applied Psychology, 90, 453–467. Ilies, R., Judge, T. A., & Wagner, D. T. (2010). The influence of cognitive and affective

reactions to feedback on subsequent goals: Role of behavioral inhibition/activation. European Psychologist, 15, 121.

REFERENCES

92

Jackson, D. N. (1984). Multidimensional aptitude battery. Port Huron, MI: Research Psychologists Press.

Jackson, D. N. (1994). Jackson Personality Inventory-revised. Sigma Assessment Systems,

Research Psychologists Press Division. Jackson, D. N., Paunonen, S. V., Fraboni, M., & Goffin, R. D. (1996). A five-factor versus six-

factor model of personality structure. Personality and Individual Differences, 20, 33-45.

Jackson, C. J. (2003). Gray's reinforcement sensitivity theory: A psychometric critique.

Personality and Individual Differences, 34, 533-544. John, O., & Srivastava, S. (1999). The Big Five Trait taxonomy: History, measurement, and

theoretical perspectives. In Pervin, L. A., & John, O. P. (Eds.), Handbook of Personality: Theory and Research (2nd ed.) (pp. 102-138). New York: Guilford Press.

Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral

economics. The American Economic Review, 93, 1449-1475. Kahneman, D., Lovallo, D., & Sibony, O. (2011). Before you make that big decision. Harvard

Business Review, 89, 50-60. Kauer, D., Waldeck, T., & Schäffer, U. (2007). Effects of top management team

characteristics on strategic decision-making: Shifting attention to team member personalities and mediating processes. Management Decision, 45, 942-967. doi:10.1108/00251740710762017

Kliger, D., & Kudryavtsev, A. (2010). The availability heuristic and investors' reaction to

company-specific events. The Journal of Behavioral Finance, 50-65. Kliger, D., & Levit, B. (2009). Evaluation periods and asset prices: Myopic loss aversion at

the financial marketplace. Journal of Economic Behavior and Organization, 71, 361–371.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A

historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.

Kluger, A. N., & DeNisi, A. (1998). Feedback interventions: Toward the understanding of a

double-edged sword. Current Directions in Psychological Science, 7, 67-72. Kouzes, J., & Posner, B. (2014, February 27). To get honest feedback, leaders need to ask

[Web log message]. Retrieved from https://hbr.org/2014/02/to-get-honest-feedback-leaders-need-to-ask/

REFERENCES

93

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficultires in recognizing one’s own incopmetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134.

Kruger, J., Wirtz, D., Van Boven, L., & Altermatt, T. W. (2003). The effort heuristic. Journal of

Experimental Social Psychology, 91-98. Latham, G. P., & Locke, E. A. (2007). New developments in and directions for goal-setting

research. European Psychologist, 12, 290-300. Lee, K., & Ashton, M. C. (2004). Psychometric properties of the HEXACO Personality

Inventory. Multivariate Behavioral Research, 39, 329-358. Leonard, D., & McAdam, R. (2002). The role of the business excellence model in operational

and strategic decision-making. Management Decision, 40, 17-25. LePine, J. A., Erez, A., & Johnson, D. E. (2002). The nature and dimensionality of

organizational citizenship behavior: a critical review and meta-analysis. Journal of Applied Psychology, 87, 52.

Levin, I. P., Gaeth, G. J., Schreiber, J., & Lauriola, M. (2002). A new look at framing effects:

Distribution of effect sizes, individual differences, and independence of types of effects. Organizational Behavior and Human Decision Processes, 88(1), 411-429. doi:10.1006/obhd.2001.2983

Lobello, S. G. (1996). Review of the Hogan Personality Inventory (Revised). In J. C. Impara &

J. C. Conoley (Eds.), The Supplement to the Twelfth Mental Measurements Yearbook. Lincoln: The University of Nebraska Press.

Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance.

Englewood Cliffs, NJ: Prentice Hall. Lovallo, D., & Sibony, O. (2010). The case for behavioral strategy. Mckinsey quarterly.

Retrieved from http://www.mckinsey.com/insights/strategy/the_case_for_behavioral_strategy

Marsden, K. (1998). The relationship between self-efficacy, intrinsic motivation, and

performance following negative feedback: A study of collegiate swimmers. Dissertation Abstracts International, 58.

McGhee, R. L., Ehrler, D. J., Buckhalt, J. A., & Phillips, C. (2012). The relation between five-

factor personality traits and risk-taking behavior in preadolescents. Psychology, 3, 558-561. doi:10.4236/psych.2012.38083

McGraw, A. P., Larsen, J. T., Kahneman, D., & Schkade, D. (2010). Comparing gains and

losses. Psychological Science, 21, 1438–1445.

REFERENCES

94

McIntyre, D. A. (2012). The worst business decisions of all time. Retrieved from http://247wallst.com/2012/10/17/the-worst-business-decisions-of-all-time/.

McNaught, J. E. (2013). How Baby Boomers experienced leaders use intuition in decision-

making. Dissertation Abstracts International Section A, 73. McPhail, S. M. (2007). Alternative validation strategies. San Francisco: Jossey-Bass. Milkman, K. L., Chugh, D., & Bazerman, M. H. (2009). How can decision-making be

improved? Perspectives on Psychological Science, 4, 379-383. doi:10.1111/j.1745-6924.2009.01142.x

Mintzberg, H., & Westley, F. (2001). Decision-making: It's not what you think. MIT Sloan

Management Review, 42, 89-93. Mol, S. T., Born, M. P., Willemsen, M. E., & Van Der Molen, H. T. (2005). Predicting Expatriate

Job Performance for Selection Purposes A Quantitative Review. Journal of Cross-Cultural Psychology, 36, 590-620.

Nutt, P. C. (1999). Surprising but true: Half the decisions in organizations fail. The Academy

of Management Executive, 13, 75-90. Nutt, P. C. (2004). Expanding the search for alternatives during strategic decision-

making. The Academy of Management Executive, 18, 13-28. O'Connell, G., Christakou, A., Haffey, A. T., & Chakrabarti, B. (2013). The role of empathy in

choosing rewards from another's perspective. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00174

Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional

predictors of organizational citizenship behavior. Personnel Psychology, 48, 775-802. Pachur, T., & Hertwig, R. (2006). On the psychology of the recognition heuristic: retrieval

primacy as a key determinant of its use. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 983.

Parker, A. M., & Fischhoff, B. (2005). Decision-making competence: External validation

through an individual-differences approach. Journal of Behavioral Decision-making, 18, 1–27.

Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker.

Cambridge University Press. Pellegrino, K. (1996). Strategic thinking ability: Cognitive and personality

effects. Dissertation Abstracts International Section A: Humanities and Social Sciences, 57.

REFERENCES

95

Peeters, M. A., Van Tuijl, H. F., Rutte, C. G., & Reymen, I. M. (2006). Personality and team performance: a meta-‐analysis. European Journal of Personality, 20, 377-396.

Rieskamp, J., & Otto, P. E. (2006). SSL: a theory of how people learn to select

strategies. Journal of Experimental Psychology: General, 135, 207. Rozin, P., & Royzman, E. B. (2001). Negativity bias, negativity dominance, and contagion.

Personality and Social Psychology Review, 5, 296–320. Rubinstein, A. (1998). Modeling bounded rationality (Vol. 1). MIT press. Russ, F. A., McNeilly, K. M., & Comer, J. M. (1996). Leadership, decision-making and

performance of sales managers: A multi-level approach. Journal of Personal Selling and Sales Management, 16, 1-15.

Russell, M., & Karol, D. (2002). 16PF Fifth Edition with updated norms administrator’s

manual. Champaign, IL: Institute for Personality and Ability Testing, Inc. Safi, A., & Burrell, D. N. (2007). Developing Advanced Decision-making Skills in International

Leaders and Managers. Vikalpa, 32, 1-8. Salmoni, A. W., Schmidt, R. A., & Walter, C. B. (1984). Knowledge of results and motor

learning: a review and critical reappraisal. Psychological Bulletin, 95, 355. Saucier, G. (1994). Mini-markers: A brief version of Goldberg’s unipolar Big-Five markers.

Journal of Personality Assessment, 63, 506-516. Schmidt, J. B., & Calantone, R. J. (2002). Escalation of commitment during new product

development. Journal of Academy of Marketing Science, 103-118. Shah, A. K., & Oppenheimer, D. M. (2008). Heuristics made easy: An effort-reduction

framework. Psychological Bulletin, 207-222. Shrauger, J. S., & Rosenberg, S. E. (1970). Self-‐esteem and the effects of success and

failure feedback on performance1. Journal of Personality, 38, 404-417. Simon, H. A. (1979). Rational decision-making in business organizations. The American

Economic Review, 69, 493-513. Smither, J. W., London, M., & Reilly, R. R. (2005). Does performance improve following

multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology, 58, 33-66. doi:10.1111/j.1744-6570.2005.514_1.x

Snook, B., Taylor, P. J., & Bennell, C. (2004). Geographic profiling: The fast, frugal, and

accurate way. Applied Cognitive Psychology, 18, 105-121. Snow, C. P. (1956). The two cultures. New Statesman, 6, 413-414.

REFERENCES

96

Snow, C. P. (1959). The two cultures and the scientific revolution: The rede lecture, 1959. University Press.

Soane, E., Dewberry, C., & Narendran, S. (2010). The role of perceived costs and perceived

benefits in the relationship between personality and risk-related choices. Journal of Risk Research, 13, 303–318.

Society for Industrial and Organizational Psychology (2003). Principles for the validation and

use of personnel selection procedures (4th ed.). Bowling Green, OH: Author. Spence, A., & Townsend, E. (2008). Spontaneous evaluations: Similarities and differences

between the affect heuristic and implicit attitudes. Cognition and Emotion, 22, 83-93.

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for

the rationality debate? Behavioral and Brain Sciences, 23, 645–726. Steers, R. M. (1975). Task-goal attributes, n achievement, and supervisory

performance. Organizational Behavior and Human Performance, 13, 392-403. Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve

diagnostic decisions. Psychological Science in the Public Interest, 1, 1-26. Tang, Y., Qian, C., Chen, G., & Shen, R. (2014). How CEO hubris affects corporate social

(ir)responsibility. Strategic Management Journal, Forthcoming. Tellegen, A., & Waller, N. G. (2008). Exploring personality through test construction:

Development of the Multidimensional Personality Questionnaire. The SAGE handbook of personality theory and assessment, 2, 261-292.

Tetlock, P. E. (2005). Expert political judgment. Princeton, NJ: Princeton University Press. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and

biases. Science, 185, 1124-1131. Weber, E. U., Blais, A., & Betz, N. E. (2002). A domain-specific risk-attitude scale: Measuring

risk perceptions and risk behaviors. Journal of Behavioral Decision-making, 15, 263-290. doi:10.1002/bdm.414

Weir, S. (2005). History’s worst decisions and the people who made them. New York, NY:

EYE Quarto, Inc. Wiggins, J. S. (Ed.). (1996). The five-factor model of personality. New York: Guilford. Wilson, G. D., Barrett, P. T., & Gray, J. A. (1989). Human reactions to reward and punishment:

A questionnaire examination of Gray's personality theory. British Journal of Psychology, 80, 509-515.

REFERENCES

97

Wilson, S. M. (2010). Decision-making by school psychologists: Use of the representativeness heuristic and importance of assessment data in determination of special education eligibility (Doctoral dissertation). Retrieved from UMI. (3420304).

Zajonc, R. B. (1980). Feeling and thinking: Preferences need no inferences. American

Psychologist, 151-175. Zenger, J., & Folkman, J. (2014, March 10). Do you need to lighten up or toughen up? [Web

log message]. Retrieved from https://hbr.org/2014/03/do-‐you-‐need-‐to-‐lighten-‐up-‐or-‐toughen-‐up

Zuckerman, M. (2007). Sensation seeking and risky behavior. Washington, DC: American

Psychological Association. Zhu, D. H., & Chen, G. (2014). CEO narcissism and the impact of interlocks on corporate

strategy. Administrative Science Quarterly, Forthcoming.

APPENDIX A: SAMPLE HOGAN JUDGMENT REPORT

98

APPENDIX A: Sample Hogan Judgment Report


99

APPENDIX A: Sample Hogan Judgment Report (Continued)


100



101



102



103



104



105



106


APPENDIX B: COMPLETE CORRELATION MATRICES

107

APPENDIX B: Complete Correlation Matrices

Table B.1 Correlations Between Judgment Scales and HPI Scales & Subscales 1 2 3 4 5 6 7 8 9 Adjustment .08 .01 -.03 .76** -.02 .08 .56** .02 .08 Empathy -.10 -.06 .05 .67** .10 -.04 .50** -.10 .02 Not Anxious .23* .11 .09 .46** .01 .07 .37** -.07 -.04 No Guilt .10 -.01 -.14 .48** -.18 .22* .37** .17 .16 Calmness .13 .02 -.17 .49** -.11 .14 .34** -.04 -.09 Even Tempered .03 -.08 -.04 .60** .11 -.05 .44** -.07 .01 No Complaints .04 .04 .06 .50** -.10 .09 .34** .01 .05 Trusting .10 .20 .11 .29** .01 .14 .30** .09 .26* Good Attachment -.02 -.08 -.06 .33** .06 -.12 .18 .04 .02 Ambition .34** .66** .01 .29** -.58** .49** .18 .05 .13 Competitive .40** .38** -.17 .02 -.51** .36** -.06 .19 .13 Self Confidence .25* .39** .00 .26** -.31** .29** .20* -.04 .12 Accomplishment .14 .31** .10 .37** -.30** .21* .20* -.08 .02 Leadership .30** .57** -.08 .05 -.64** .47** -.02 .09 .11 Identity .11 .10 -.01 .40** -.16 .29** .37** .12 .12 No Social Anxiety .14 .63** .19 .19 -.23* .24* .17 -.09 .02 Sociability .39** .46** -.02 -.12 -.39** .26* -.14 .09 .10 Likes Parties .13 .30** .07 -.03 -.18 .13 -.05 -.16 -.13 Likes Crowds .11 .22* .09 -.06 -.21* .22* .00 .20* .08 Experience Seeking .60** .38** -.27** -.10 -.41** .34** -.08 .02 .25* Exhibitionistic .25* .29** -.02 -.16 -.23* .02 -.25* .04 .07 Entertaining .05 .20 .10 .01 -.13 .07 -.02 .16 -.01 Interpersonal Sensitivity -.04 .25* .05 .43** -.07 .12 .31** -.17 -.03 Easy to Live With -.07 .07 .03 .51** .04 -.10 .28** -.26** -.11 Sensitivity -.09 .02 -.03 .18 -.05 .08 .13 -.13 -.10 Caring -.11 .05 -.04 .28** -.04 .05 .18 .00 .03 Likes People .11 .45** .10 .14 -.25* .27** .13 -.08 .03 No Hostility -.16 -.01 -.02 .47** .21* -.12 .36** -.13 -.07 Prudence -.29** -.03 .09 .48** .11 -.09 .34** -.04 -.11 Moralistic -.13 -.07 .00 .34** .02 .00 .23* -.18 -.25* Mastery -.04 -.02 .00 .04 -.16 .10 .00 .05 -.08 Virtuous -.07 .04 .11 .51** .03 .08 .41** -.10 -.16 Not Autonomous -.07 .09 -.04 -.19 .02 -.11 -.18 -.09 -.06 Not Spontaneous -.23* -.07 .12 .05 .17 -.05 .12 .11 .03 Impulse Control -.42** -.17 .05 .26* .20 -.25* .11 .11 -.02 Avoids Trouble .10 .10 -.02 .42** .02 -.03 .27** -.01 .17 Inquisitive .57** .19 -.47** .03 -.34** .31** .00 .19 .22* Science Ability .45** .16 -.55** -.01 -.34** .32** -.02 .13 .19 Curiosity .45** .02 -.50** .02 -.22* .15 -.04 .16 .13 Thrill Seeking .41** .27** -.05 -.11 -.27** .22* -.09 -.01 .02 Intellectual Games .13 .04 -.20 -.02 -.08 .04 -.06 .33** .22* Generates Ideas .29** .30** -.05 .17 -.39** .20* .01 .13 .10 Culture .24* -.13 -.26** .06 .11 .09 .14 -.01 .09 Learning Approach .31** .20* -.33** -.08 -.27** .13 -.12 .32** .26** Education .33** .15 -.18 -.01 -.20* .12 -.02 .26* .30** Math Ability .50** .25* -.38** -.09 -.42** .20* -.19 .28** .25* Good Memory .06 .13 -.19 -.04 -.15 .04 -.06 .17 .09 Reading -.01 .03 -.14 -.08 .01 -.02 -.08 .16 .07 Note. N = 98; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


108

Table B.2 Correlations Between Judgment Scales and HDS Scales & Subscales 1 2 3 4 5 6 7 8 9 Excitable -.17 -.31** -.13 -.47** .20 -.30** -.40** .02 -.03 Volatile -.01 -.13 -.14 -.52** -.03 -.11 -.43** .08 .04 Easily Disappointed -.04 -.19 -.10 -.34** .10 -.30** -.38** -.04 .03 No Direction -.35** -.36** -.03 -.14 .39** -.23* -.02 .02 -.15 Skeptical -.03 -.26* .10 -.45** .08 -.25* -.41** .02 -.01 Cynical .02 -.09 -.02 -.38** .09 -.11 -.26* -.04 -.09 Mistrusting .08 -.22* .06 -.35** .06 -.35** -.42** .04 .07 Grudges -.14 -.25* .14 -.30** .05 -.12 -.25* .03 .00 Cautious -.35** -.62** -.06 -.17 .44** -.42** -.18 .09 .04 Avoidant -.28** -.62** .00 -.06 .48** -.39** -.04 .09 .04 Fearful -.39** -.59** -.01 -.27* .43** -.40** -.23* .18 .07 Unassertive -.15 -.27* -.13 -.06 .13 -.21* -.15 -.05 -.02 Reserved -.21 -.31** -.18 -.05 .26* -.29** -.06 .07 .09 Introverted -.27* -.35** -.18 -.07 .29** -.35** -.13 .04 .06 Unsocial -.20 -.24* -.14 -.12 .30** -.34** -.13 .15 .17 Tough -.04 -.18 -.12 .10 .03 .01 .11 -.06 -.05 Leisurely -.16 -.24* -.14 -.24* .12 -.32** -.32** -.04 -.10 Passive Aggressive -.11 -.24* .05 -.04 .17 -.28** -.14 .01 -.05 Unappreciated -.06 -.12 -.11 -.36* .03 -.22* -.36** -.16 -.02 Irritated -.16 -.14 -.28** -.13 .04 -.17 -.20 .06 -.14 Bold .17 .36** -.11 .17 -.55** .24* -.06 -.19 -.20 Entitled .08 .22* -.17 .11 -.24* .19 .04 -.19 -.06 Overconfidence .04 .21 .00 .29** -.41** .14 .02 -.16 -.22* Fantasized Talent .25* .41** -.08 .00 -.59** .22* -.20 -.09 -.17 Mischievous .42** .19 -.06 -.15 -.32** .09 -.20 -.05 -.14 Risky .47** .17 -.10 -.16 -.24* .14 -.14 -.06 -.03 Impulsive .34** .19 -.19 -.10 -.29** .13 -.15 -.03 .03 Manipulative .11 .09 .16 -.08 -.21 -.08 -.19 -.01 -.35** Colorful .47** .44** -.20 -.14 -.39** .40** -.06 .16 -.12 Public Confidence .35** .54** -.08 -.03 -.38** .37** .01 .06 -.21 Distractible .36** .15 -.17 -.23* -.18 .28** -.07 .21 -.02 Self-Display .33** .26* -.19 -.06 -.28* .22* -.09 .09 -.02 Imaginative .27* .19 -.29** -.07 -.43** .15 -.23* -.04 -.12 Eccentric .23* -.01 -.27* -.26* -.05 .10 -.15 .03 .09 Special Sensitivity .18 .22* -.13 .05 -.44** .15 -.14 -.12 -.29** Creative Thinking .29** .18 -.25* -.08 -.36** .08 -.24* -.03 .01 Diligent -.13 .00 -.06 -.14 -.10 -.15 -.27* -.11 -.11 Standards .00 .09 -.12 -.06 -.30** -.02 -.24* -.06 -.06 Perfectionistic -.15 -.04 -.14 -.28** -.01 -.22* -.36** -.09 -.11 Organized -.12 -.02 .11 .05 .02 -.07 .00 -.09 -.07 Dutiful -.38** -.26* .16 -.09 .16 -.34** -.22* -.03 -.17 Indecisive -.41** -.27* .07 .00 .12 -.18 -.09 -.03 -.10 Ingratiating -.16 -.17 .08 -.17 .14 -.31** -.26* -.02 -.06 Conforming -.33** -.17 .22* -.05 .12 -.30** -.17 -.01 -.23* Note. N = 85; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


109

Table B.3 Correlations Between Judgment Scales and CPI Scales 1 2 3 4 5 6 7 8 9 Dominance .26** .69** -.03 .02 -.53** .42** -.02 -.06 .00 Capacity for Status .36** .55** -.14* .11 -.35** .38** .12 .02 .13 Sociability .31** .51** .01 .07 -.34** .32** .07 -.03 .03 Social Presence .39** .38** -.11 .07 -.25** .25** .06 -.01 .09 Self-Acceptance .41** .67** -.06 .01 -.51** .43** .00 .06 .11 Independence .35** .55** -.10 .22** -.36** .44** .22** -.04 .06 Empathy .32** .37** -.06 .15* -.21** .26** .15* .03 .03 Responsibility -.02 .17* -.13 .27** -.07 .27** .29** .13 .10 Socialization -.14 -.06 -.03 .37** .04 .03 .27** .09 .01 Self-Control -.24** -.09 -.01 .47** .17* .15* .49** .01 -.04 Good Impression -.14 .09 -.06 .42** -.01 .17* .37** -.08 -.04 Communality .11 .16* -.09 .16* -.18* .25** .15* .06 .06 Well-Being .07 .14 -.11 .53** -.07 .33** .51** -.02 -.03 Tolerance .01 .04 -.06 .35** .14 .23** .44** .12 .09 Achievement via Conformance .01 .34** -.11 .32** -.25** .28** .25** .07 .06 Achievement via Independence .24** .27** -.29** .17* -.06 .31** .26** .11 .18* Intellectual Efficiency .34** .36** -.29** .15* -.15* .41** .26** .16* .19** Psychological-Mindedness .19** .23** -.28** .23** -.10 .28** .25** .07 .08 Flexibility .24** .04 -.16* .12 .07 .12 .19** .05 .01 Femininity/Masculinity -.35** -.24** .24** -.11 .29** -.26** -.09 .05 -.08 Externality/Internality -.32** -.65** .01 .11 .56** -.30** .19** .04 -.04 Norm-Doubting/Norm-Favoring -.11 .10 -.03 .15* -.18* .05 .02 .05 .03 Ego-Integration .15* .23** -.21** .39** -.03 .34** .44** .08 .10 Note. N = 195; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


110

Table B.4 Correlations Between Judgment Scales and NEO PI-R Scales & Facets 1 2 3 4 5 6 7 8 9 Neuroticism -.12 -.29** .06 -.49** .15* -.34** -.44** .12 -.04 Anxiety -.19** -.21** .03 -.38** .12 -.29** -.35** .13 -.06 Angry Hostility .01 -.08 .09 -.60** -.05 -.11 -.47** .10 .03 Depression -.08 -.27** .03 -.37** .15* -.27** -.32** .09 -.08 Self-Consciousness -.27** -.42** .04 -.26** .22** -.41** -.30** .07 -.08 Impulsiveness .08 -.09 .03 -.34** .02 -.21** -.33** .05 -.01 Vulnerability -.10 -.29** .08 -.31** .25** -.28** -.23** .12 .03 Extraversion .21** .44** .17* -.02 -.36** .23** -.07 -.03 .06 Warmth -.06 .19* .20** .12 -.10 .06 .05 -.06 -.01 Gregariousness .08 .27** .21** .03 -.14 .16* .05 -.05 -.03 Assertiveness .18* .67** .13 -.03 -.52** .32** -.12 -.03 .01 Activity .15* .36** .02 -.12 -.33** .20** -.13 .11 .14 Excitement-Seeking .33** .11 -.01 -.19** -.23** .07 -.21** -.06 .10 Positive Emotions .12 .13 .11 .10 -.07 .08 .07 -.02 .04 Openness .31** .11 -.33** -.19** -.11 .13 -.11 .07 .12 Fantasy .26** .10 -.22** -.18* -.14 .04 -.17* .11 .10 Aesthetics .16* .01 -.17* -.16* .00 .08 -.06 .03 .00 Feelings .05 .06 .03 -.30** -.07 -.02 -.24** .08 .08 Actions .39** .14 -.17* .03 -.12 .24** .10 .00 .01 Ideas .24** .19** -.44** -.18* -.24** .14 -.19* .06 .19** Values .12 -.06 -.27** .05 .13 .04 .13 .00 .07 Agreeableness -.29** -.17* .04 .28** .29** -.06 .30** -.13 -.09 Trust -.10 .11 .03 .33** .08 .10 .31** -.13 -.07 Straightforwardness -.27** -.12 .00 .13 .23** .05 .26** -.09 -.01 Altruism -.23** .01 .06 .18* .01 -.03 .09 -.04 -.07 Compliance -.22** -.17* .01 .39** .25** -.13 .30** -.13 -.14 Modesty -.23** -.36** -.02 .00 .35** -.14 .12 -.01 -.04 Tender-Mindedness -.11 -.15* .09 .05 .20** -.10 .07 -.09 -.05 Conscientiousness -.06 .27** -.03 .11 -.33** .18* -.01 .03 .03 Competence .02 .28** -.09 .18* -.31** .20** .05 .02 .10 Order -.04 .09 .12 -.07 -.14 .10 -.05 .02 -.01 Dutifulness -.20** .14 .05 .04 -.20** .01 -.08 -.02 .09 Achievement Striving .14 .40** -.15* -.02 -.45** .20** -.15* .01 -.02 Self-Discipline -.02 .29** .04 .16* -.32** .24** .08 .03 .02 Deliberation -.20** -.04 -.12 .21** .02 .00 .13 .08 -.01 Note. N = 182; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.5 Correlations Between Judgment Scales and IPIP Big 5 20-Item Scales 1 2 3 4 5 6 7 8 9 Extraversion .21** .52** .18* -.02 -.34** .26** -.05 -.07 .02 Agreeableness .02 .11 .10 .13 -.08 .16* .13 -.03 .01 Conscientiousness -.11 .18* .00 .00 -.24** .20** .01 -.07 -.03 Emotional Stability .14 .14 -.08 .57** .00 .18* .50** -.16* -.05 Intellect / Imagination .32** .35** -.31** -.12 -.35** .25** -.13 .15* .28** Note. N = 169; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


111

Table B.6 Correlations Between Judgment Scales and 16PF Scales 1 2 3 4 5 6 7 8 9 Warmth .00 .21** .37** .06 -.06 .12 .09 -.11 -.04 Reasoning .12 .09 -.26** -.01 -.04 .14* .05 .28** .41** Emotional Stability .11 .19** .02 .38** -.15* .31** .37** -.03 .00 Dominance .31** .48** -.08 -.22** -.43** .32** -.17* -.08 .07 Liveliness .12 .15* .12 -.11 -.17* .01 -.15* -.03 .07 Rule-Consciousness -.18* .06 .12 .14* -.07 .06 .10 -.05 -.09 Social-Boldness .12 .59** .20** .00 -.37** .25** -.04 -.12 -.04 Sensitivity -.16* -.06 .25** -.01 .16* -.12 -.02 .02 .03 Vigilance -.04 -.15* -.04 -.20** .01 -.18* -.24** -.04 -.09 Abstractedness .24** .05 -.23** -.20** -.10 -.08 -.25** .03 .07 Privateness .02 -.25** -.25** .04 .14 -.22** -.05 .07 .03 Apprehension -.27** -.27** .03 -.29** .14 -.34** -.35** -.03 -.11 Openness to Change .36** .28** -.24** -.16* -.18* .26** -.05 -.03 .11 Self-Reliance -.11 -.13 -.07 -.06 .07 -.24** -.15* -.09 -.04 Perfectionism -.18* .06 .05 -.08 -.21** .14* -.07 -.02 .05 Tension .00 -.07 -.07 -.28** .00 -.12 -.28** .09 .16* Note. N = 197; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.7 Correlations Between Judgment Scales and TCI Scales 1 2 3 4 5 6 7 8 9 Exploratory Excitability vs. Stoic Rigidity .51** .26** -.14* -.09 -.26** .20** -.08 .02 .09 Impulsiveness vs. Reflection .26** .12 .19** .03 .01 .06 .06 -.08 -.01 Extravagance vs. Reserve .05 .02 .12 -.14* .04 -.03 -.07 .06 .07 Disorderliness vs. Regimentation .24** .16* -.06 -.07 -.22** .00 -.15* .05 .07 Pessimism vs. Optimism -.27** -.27** .03 -.29** .26** -.40** -.31** .11 -.02 Fear of Uncertainty -.50** -.27** .10 -.04 .27** -.27** -.05 .15* -.02 Shyness with Strangers -.27** -.53** -.08 -.12 .37** -.30** -.06 .10 .02 Fatigability and Asthenia -.22** -.21** .12 -.19** .13 -.21** -.18** .02 -.07 Sentimentality -.14* -.05 .21** -.04 .01 -.11 -.09 -.03 .03 Openness to Communication vs. Aloofness .07 .34** .24** -.01 -.22** .26** .05 -.06 -.01 Attachment .04 .22** .17* -.04 -.09 .25** .09 -.04 .00 Dependence -.29** -.11 .20** .17* .19** -.14* .11 .01 -.08 Eagerness of Effort vs. Laziness .06 .28** .00 .05 -.32** .23** .02 -.09 -.07 Work-Hardened vs. Spoiled .20** .24** -.17* .04 -.38** .26** -.02 .02 .05 Ambitious vs. Underachieving .14* .44** -.14* -.04 -.53** .18* -.20** -.02 -.02 Perfectionist vs. Pragmatist .07 .28** -.20** -.06 -.36** .15* -.15* -.04 .00 Responsibility vs. Blaming .22** .24** .03 .29** -.19** .28** .26** .07 .05 Purposefulness vs. Lack of Goals .09 .38** -.11 .28** -.29** .27** .19** -.03 .05 Resourcefulness .33** .48** -.16* .10 -.44** .45** .11 -.01 .06 Self-Acceptance vs. Self-Striving .11 -.02 .04 .30** .16* .24** .43** -.01 .01 Enlightened Second Nature -.07 .13 .02 .28** -.11 .16* .22** -.04 -.02 Social Acceptance vs. Social Intolerance .08 .05 .08 .22** .07 .11 .26** -.01 -.03 Empathy vs. Social Disinterest .04 .10 .05 .08 -.10 .06 .02 -.07 -.04 Helpfulness vs. Unhelpfulness .02 .16* .01 .18* -.05 .20** .23** .10 .04 Compassion vs. Revengefulness .07 .12 .05 .27** -.01 .18* .28** -.09 -.07 Conscience vs. Self-Serving Advantage -.02 .14* .05 .04 -.02 .12 .09 .13 .13 Self-Forgetful vs. Self-Conscious .21** .12 -.10 -.16* -.29** .04 -.26** -.03 -.02 Identification vs. Self-Differentiation .12 .12 -.04 -.07 -.21** .07 -.13 -.15* -.09 Spiritual Acceptance vs. Materialism .05 .09 .30** -.09 -.11 .00 -.10 -.05 .02 Enlightened vs. Objective .00 .05 .34** -.06 -.10 .00 -.08 -.01 -.03 Idealistic vs. Practical -.03 .06 .30** -.05 -.10 -.03 -.10 -.04 -.04 Note. N = 203; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


112

Table B.8 Correlations Between Judgment Scales and MPQ Scales 1 2 3 4 5 6 7 8 9 Well-Being .16* .24** -.07 .21** -.32** .16* .05 -.08 .03 Social Potency .27** .62** .06 -.16* -.57** .26** -.25** -.08 -.03 Achievement .18* .23** -.28** -.04 -.36** .17** -.12 .04 .06 Social Closeness .04 .26** .16* .12 -.14* .21** .13 .01 .02 Stress Reaction -.13 -.27** .01 -.51** .18** -.30** -.43** .09 -.02 Aggression -.01 -.12 -.01 -.36** .02 -.02 -.22** .11 .02 Alienation .02 -.07 -.03 -.28** .03 -.06 -.19** -.08 -.07 Control -.29** -.03 .01 .17* .04 .01 .14* .10 .07 Harm Avoidance -.36** -.12 .15* .16* .12 -.12 .10 .14* -.06 Traditionalism -.19** .04 .25** .02 -.17* -.07 -.11 -.09 -.14 Absorption .16* -.05 -.11 -.26** -.11 -.11 -.32** -.06 -.04 Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.9 Correlations Between Judgment Scales and 6FPQ Factors & Facets 1 2 3 4 5 6 7 8 9 Extraversion Factor .26** .58** .12 -.04 -.50** .35** -.07 -.03 -.03 Affiliation Facet .08 .34** .19** .06 -.25** .18* .02 -.03 -.03 Dominance Facet .26** .52** -.07 -.04 -.49** .38** -.05 -.02 .00 Exhibition Facet .26** .50** .18* -.09 -.41** .24** -.13 -.03 -.05 Agreeableness Factor -.06 -.10 .06 .52** .19** -.09 .38** -.11 -.10 Abasement Facet .05 -.01 .04 .26** .07 -.09 .14 -.06 -.07 Even-Tempered Facet -.15* -.13 .04 .59** .18* -.05 .46** -.11 -.10 Good-Natured Facet -.04 -.11 .07 .40** .21** -.08 .32** -.11 -.06 Independence Factor .17* -.08 -.19** .07 .02 .07 .10 .03 .11 Autonomy Facet .22** -.02 -.22** -.01 -.05 .02 -.03 .04 .13 Individualism Facet .06 -.15* -.06 .19** .16* .12 .29** .01 .05 Self-Reliance Facet .11 -.01 -.17* -.02 -.07 .02 -.04 .01 .08 Openness to Experience Factor .34** .24** -.30** -.10 -.16* .17* -.07 .05 .10 Change Facet .37** .14 -.12 -.06 -.12 .08 -.07 -.02 .03 Understanding Facet .16* .17* -.30** -.10 -.10 .06 -.12 .09 .11 Breadth of Interest Facet .28** .26** -.26** -.06 -.16* .27** .03 .03 .10 Methodicalness Factor -.29** -.04 .06 .06 -.02 .03 .06 -.06 -.04 Cognitive Structure Facet -.32** -.07 .08 .04 .07 -.03 .05 -.10 -.06 Deliberateness Facet -.23** -.05 -.11 .23** -.03 -.01 .11 .02 -.01 Order Facet -.16* .02 .13 -.07 -.07 .09 -.01 -.06 -.03 Industriousness Factor .01 .20** -.27** .01 -.24** .17* -.01 -.04 .00 Achievement Facet .07 .24** -.11 -.04 -.24** .11 -.08 .02 .01 Endurance Facet .15* .17* -.32** .02 -.26** .25** .04 -.06 .03 Seriousness Facet -.17* .05 -.16* .04 -.05 .01 .00 -.03 -.05 Note. N = 200; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


113

Table B.10 Correlations Between Judgment Scales and PRS Scales 1 2 3 4 5 6 7 8 9 BIS Anxiety Scale -.24** -.23** .08 -.30** .16* -.31** -.32** .11 -.04 BAS Reward Responsiveness .06 .08 .12 -.14* -.15* -.02 -.20** .09 -.03 BAS Drive .19** .16* -.07 -.18** -.29** .07 -.26** .01 -.02 BAS Fun Seeking .38** .13 -.02 -.28** -.22** .04 -.29** -.01 .10 HEXACO Sincerity Scale .02 .01 .00 .22** .06 .27** .36** .02 -.03 HEXACO Fairness Scale -.16* .10 .10 .25** -.05 .10 .19** .07 .02 HEXACO Greed Avoidance Scale -.03 -.18* .02 .18* .29** .02 .28** .05 .01 HEXACO Modesty Scale -.22** -.26** .28** .21** .41** -.16* .28** .03 .02 HEXACO Fearfulness Scale -.44** -.18* .18** -.01 .24** -.19** .00 .02 -.10 HEXACO Anxiety Scale -.21** -.12 .02 -.39** .02 -.28** -.42** -.05 -.13 HEXACO Dependence Scale -.10 .00 .06 -.14* .03 -.02 -.11 -.03 .01 HEXACO Sentimentality Scale -.18* -.05 .20** -.06 .01 -.11 -.11 -.02 -.06 HEXACO Expressiveness Scale .10 .37** .16* -.30** -.26** .13 -.24** .05 .05 HEXACO Social Boldness Scale .23** .69** .06 .00 -.51** .39** -.03 -.06 -.05 HEXACO Sociability Scale .09 .24** .15* -.06 -.23** .10 -.11 -.05 .01 HEXACO Liveliness Scale .22** .33** -.02 .11 -.25** .22** .08 -.01 .12 HEXACO Forgiveness Scale .05 .00 .04 .34** .09 .03 .28** -.04 .05 HEXACO Gentleness Scale -.13 -.28** .02 .39** .23** -.16* .28** -.13 -.13 HEXACO Flexibility Scale -.05 -.16* .07 .36** .21** -.10 .30** -.08 -.02 HEXACO Patience Scale -.01 -.05 .01 .54** .06 -.04 .33** -.11 -.03 HEXACO Organization Scale -.08 .12 .23** -.01 -.14* .10 -.01 -.07 -.09 HEXACO Diligence Scale .18* .34** -.21** -.02 -.43** .26** -.09 .00 .04 HEXACO Perfectionism Scale -.12 .02 -.14* -.21** -.11 .00 -.20** .03 -.03 HEXACO Prudence Scale -.16* .06 -.12 .37** -.09 .09 .23** .10 .01 HEXACO Aesthetic Appreciation Scale .14* .05 -.20** -.03 .04 .06 .01 .01 .07 HEXACO Inquisitiveness Scale .26** .13 -.52** -.03 -.14* .08 -.08 .03 .12 HEXACO Creativity Scale .41** .28** -.23** -.17* -.32** .19** -.18* .05 .07 HEXACO Unconventionality Scale .34** .13 -.36** -.20** -.16* .12 -.17* .04 .16* HEXACO Honesty-Humility Scale -.13 -.12 .14* .29** .25** .07 .38** .05 .01 HEXACO Emotional Stability Scale -.32** -.12 .16* -.21** .11 -.21** -.22** -.03 -.09 HEXACO Extraversion Scale .21** .55** .12 -.10 -.42** .28** -.11 -.02 .04 HEXACO Agreeableness Scale -.04 -.15* .04 .52** .18** -.08 .38** -.11 -.04 HEXACO Conscientiousness Scale -.07 .20** -.05 .03 -.28** .16* -.03 .00 -.04 HEXACO Openness to Experience Scale .38** .19** -.43** -.14* -.19** .15* -.14* .04 .14* Note. N = 208; BIS = Behavioral Inhibition System; BAS = Behavioral Activation System; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


114

Table B.11 Correlations Between Judgment Scales and MVPI Scales & Themes 1 2 3 4 5 6 7 8 9 Aesthetics .20* -.02 -.16 -.29** .00 .05 -.16 .09 .04 Lifestyle .20* -.04 -.17 -.10 .07 .07 .02 .13 .16 Beliefs .09 -.15 -.14 -.26** .08 .09 -.04 .04 -.06 Occupational Preferences .19 .01 -.18 -.20* .00 -.08 -.18 .02 .00 Aversions -.01 .02 -.17 -.29** -.14 .12 -.20* .10 .09 Preferred Associates .24* .05 .03 -.28** -.05 .02 -.21* .06 .01 Affiliation .19 .28** .03 .18 -.18 .12 .06 -.15 -.02 Lifestyle .01 .26* .02 .15 -.12 .13 .10 -.07 -.02 Beliefs .15 .08 -.04 .15 -.01 -.02 .05 -.19 .24* Occupational Preferences .20* .22* .04 .26** -.17 .21* .19 -.14 -.07 Aversions .20* .16 .02 -.13 -.16 .00 -.20* -.17 -.17 Preferred Associates .09 .14 .04 .16 -.13 -.03 -.01 .03 .05 Altruistic .00 .11 .00 .16 -.04 -.03 .03 -.18 -.01 Lifestyle .03 .15 .06 .19 -.09 .00 .05 -.21* .03 Beliefs -.03 .01 .03 .18 .02 -.02 .09 -.08 .07 Occupational Preferences -.01 .03 .01 .15 .02 -.14 -.02 -.20* -.09 Aversions -.05 .00 -.10 -.08 -.03 -.05 -.12 -.10 -.01 Preferred Associates .06 .19 .08 .26** -.10 .09 .13 -.02 .07 Commercial .17 .21* -.20* -.09 -.39** .16 -.20* -.04 .02 Lifestyle .22* .23* -.28** .02 -.32** .21* -.07 -.01 .02 Beliefs -.06 -.04 .12 .01 -.19 -.10 -.14 -.05 -.09 Occupational Preferences .20 .14 -.20 -.12 -.28** .11 -.18 .03 .11 Aversions -.11 .15 .04 -.15 -.26** .08 -.19 -.11 -.10 Preferred Associates .13 .13 -.17 -.04 -.20* .13 -.07 -.04 .07 Hedonistic .13 -.04 .02 -.36** -.21* -.12 -.42** .07 .16 Lifestyle .20* -.05 .00 -.20* -.19 .04 -.19 .10 .21* Beliefs .13 -.06 -.12 -.28** -.18 -.06 -.34** .02 .02 Occupational Preferences .10 .01 .11 -.27** -.18 -.14 -.37** .05 .18 Aversions -.04 -.07 .06 -.22* -.07 -.12 -.26** .08 .14 Preferred Associates .06 .08 .04 -.23* -.20* -.11 -.34** .05 .10 Power .27** .41** -.20* -.12 -.58** .36** -.17 -.05 .01 Lifestyle .26* .40** -.23* -.08 -.53** .32** -.14 -.13 -.09 Beliefs .15 .20* -.16 -.04 -.44** .23* -.11 -.13 -.04 Occupational Preferences .40** .47** -.18 -.05 -.61** .36** -.15 .14 .18 Aversions .00 .22* -.09 -.19 -.28** .19 -.16 -.02 .04 Preferred Associates .19 .15 -.08 -.08 -.24* .17 -.06 -.06 -.04 Recognition .12 .25* -.04 -.30** -.41** -.01 -.45** .12 .15 Lifestyle .07 .22* .01 -.27** -.32** -.11 -.42** .19 .15 Beliefs .16 .12 -.02 -.25* -.22* -.02 -.28** .00 .00 Occupational Preferences .17 .33** -.07 -.09 -.45** .05 -.31** .02 .17 Aversions -.07 .01 .04 -.36** -.18 -.05 -.37** .10 .11 Preferred Associates .16 .25* -.13 .07 -.33** .16 -.08 .11 .13 Scientific .10 .05 -.67** -.04 -.11 .12 -.05 .09 .04 Lifestyle .24* .03 -.48** -.04 -.07 .15 .00 .09 .03 Beliefs -.13 -.14 -.26* .03 .07 -.11 -.01 .03 -.09 Occupational Preferences .25* .13 -.72** .12 -.18 .20* .06 .04 .10 Aversions -.12 .02 -.33** -.27** -.08 -.01 -.25* .12 .07 Preferred Associates .12 .15 -.49** .06 -.18 .20* .03 .01 .04 Note. N = 99; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


115

Table B.11 (Continued) Correlations Between Judgment Scales and MVPI Scales & Themes 1 2 3 4 5 6 7 8 9 Security -.57** -.28** .03 .04 .23* -.26** .01 -.15 -.14 Lifestyle -.38** -.05 -.04 .12 .04 -.03 .07 -.08 -.07 Beliefs -.37** -.24* -.12 .07 .07 -.10 .03 -.12 -.04 Occupational Preferences -.35** -.21* .00 .12 .34** -.28** .08 -.09 -.05 Aversions -.54** -.25* .09 -.10 .23* -.33** -.10 -.13 -.14 Preferred Associates -.28** -.21* .14 -.03 .11 -.19 -.05 -.09 -.15 Tradition -.10 .22* .13 .09 -.15 .17 .09 -.04 .10 Lifestyle -.09 .15 .14 .13 -.08 .13 .13 -.16 .01 Beliefs -.01 .18 .13 .05 -.18 .21* .10 -.03 .01 Occupational Preferences -.18 .12 .12 -.01 -.12 .09 .00 -.03 -.01 Aversions .03 .28** -.03 -.05 -.27** .20 -.10 .05 .07 Preferred Associates -.05 .11 .10 .20 .03 .03 .16 .12 .28** Note. N = 99; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.12 Correlations Between Judgment Scales and CISS Interest & Skil l Scales CISS Interest Scales 1 2 3 4 5 6 7 8 9 Influencing .26** .46** -.06 .06 -.35** .18* -.04 -.10 .01 Organizing .01 .18* -.09 .05 -.22** .10 -.01 -.04 .00 Helping -.01 .18* .08 .13 .01 .08 .13 .00 .10 Creating .09 .02 .04 -.11 .03 .00 -.06 .07 .06 Analyzing .16* .08 -.57** .07 -.21** .19* .04 .11 .15* Producing .18* -.04 -.33** .01 -.10 .15 .03 -.03 -.05 Adventuring .31** .15* -.12 .10 -.20* .15 .07 -.03 .06 CISS Skill Scales 1 2 3 4 5 6 7 8 9 Influencing .28** .47** .01 -.01 -.37** .19* -.08 -.01 .03 Organizing .15 .26** -.07 -.04 -.31** .12 -.11 .08 .12 Helping .04 .25** .19* -.01 -.10 .10 .00 .02 .11 Creating .10 .09 .07 -.12 -.08 -.01 -.15 .05 .11 Analyzing .23** .14 -.46** -.01 -.25** .20** -.03 .21** .23** Producing .21** .01 -.33** -.07 -.19* .15 -.07 -.03 .04 Adventuring .33** .13 -.14 -.05 -.20** .10 -.07 .02 .15 Note. N = 166; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


116

Table B.13 Correlations Between Judgment Scales and JPI-R Scales 1 2 3 4 5 6 7 8 9 Analytical Cluster Complexity .22** .21** -.25** -.10 -.12 .10 -.08 .20** .22** Breadth of Interest .35** .27** -.34** -.07 -.25** .22** -.09 .01 .07 Innovation .41** .41** -.30** -.08 -.48** .29** -.14* -.02 .05 Tolerance .31** .13 -.12 .19** .01 .23** .28** .04 .01 Emotional Cluster Empathy -.14* .02 .13 -.11 -.05 -.07 -.16* -.02 -.05 Anxiety -.15* -.09 .00 -.49** .02 -.26** -.50** .10 -.01 Cooperativeness -.25** -.12 .21** -.147* .05 -.33** -.29** -.04 -.18** Extroverted Cluster Sociability .12 .22** .11 .03 -.20** .11 -.03 -.05 -.04 Social Confidence .28** .66** .05 .01 -.49** .37** -.03 -.08 -.04 Energy Level .28** .33** -.21** .03 -.30** .29** .03 -.09 -.03 Opportunistic Cluster Social Astuteness .16* .23** .02 -.15* -.22** -.07 -.27** -.01 -.02 Risk Taking .57** .30** -.18** -.10 -.32** .24** -.09 -.08 .07 Dependable Cluster Organization -.23** .06 .08 .03 -.15* .09 -.01 .00 .03 Traditional Values -.15* .04 .32** .08 -.11 .00 .01 -.02 -.15* Responsibility -.13 .09 .05 .14 -.04 .01 .05 -.04 -.03 Note. N = 207; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.14 Correlations Between Judgment Scales and HBRI Scales 1 2 3 4 5 6 7 8 9 Tactical Thinking .10 -.01 -.16* .03 .01 .06 .05 .45** .36** Strategic Thinking .19** .09 -.26** .07 -.10 .15* .07 .36** .41** Overall Critical Reasoning .15* .03 -.21** .06 -.05 .12 .08 .47** .45** Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level. Table B.15 Correlations Between Judgment Scales and SPA Cognitive Abil it ies 1 2 3 4 5 6 7 8 9 Verbal Ability .16* .19** -.08 -.05 -.17* .06 -.11 .16* .08 Mathematical Ability .22** .08 -.25** -.00 -.21** .16* -.03 .17* .20** Note. N = 211; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


117

Table B.16 Correlations Between Judgment Scales and BRI Cognitive Content 1 2 3 4 5 6 7 8 9 Reading Cluster .08 .15* -.20** .06 -.10 .05 .00 .15* .13 Music/Drama Cluster .12 .16* .02 -.01 -.07 .07 -.01 .05 .17* Political/Organizational Cluster .22** .27** -.09 .09 -.14* .20** .08 .07 .14* Books Cluster .13 .07 -.14 -.02 .06 -.02 .00 .15* .20** Culture Cluster .20** .15* -.19** .07 -.04 .09 .06 .04 .22** Computing Cluster .12 .13 -.16* .08 -.08 .03 .02 .21** .25** Work Cluster .15* .17* -.14* -.06 -.07 .19** .07 .32** .34** Creative Achievements Cluster .23** .25** -.11 -.13 -.15* .17* -.06 .12 .24** Note. N = 205; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


118

Table B.17 Judgment Scale Correlations with Observer Adjective Ratings 1 2 3 4 5 6 7 8 9 Bashful -.19** -.37** -.09 -.06 .24** -.26** -.06 .03 -.04 Bold .15* .42** .14 -.05 -.29** .28** .00 -.07 -.07 Careless .11 .07 .03 -.06 .00 -.03 -.06 -.09 .00 Cold .01 .09 -.05 -.04 -.20** .02 -.13 -.01 .04 Complex .21** .26** -.31** -.18* -.28** .12 -.22** .10 .07 Cooperative -.13 -.10 -.03 .22** .11 .01 .21** -.04 -.10 Creative .14 .14* -.09 -.07 -.14 -.01 -.14 .04 .05 Deep .13 .15* -.14 -.07 -.11 .11 -.05 .17* .13 Disorganized .16* .04 -.12 .03 .01 -.03 .01 -.22** -.07 Efficient -.13 .09 .04 -.03 -.14 .11 -.02 .17* .03 Energetic .15* .24** .03 -.06 -.20** .18* -.04 -.02 .02 Envious .11 .06 .01 -.17* -.01 -.06 -.15* -.05 -.03 Extraverted .08 .31** .14* -.05 -.20** .13 -.05 -.11 -.04 Fretful -.10 -.12 .01 -.22** .06 -.21** -.24** -.07 -.11 Good-Looking -.09 -.02 .20** .02 .07 .05 .12 .04 .03 Harsh .11 .12 -.02 -.23** -.11 -.03 -.25** -.07 .02 Imaginative .19** .20** -.12 -.04 -.14 .11 -.03 .00 .06 Inefficient .12 -.11 -.12 -.02 .10 -.11 -.04 -.07 -.07 Intellectual .12 .24** -.18* -.02 -.25** .10 -.10 .17* .22** Jealous .03 .08 .06 -.24** -.05 -.08 -.25** -.07 -.11 Kind -.09 -.03 .07 .19** .09 -.07 .13 -.06 .03 Moody -.02 -.07 .03 -.26** .06 -.15* -.23** .01 -.04 Organized -.14* .01 .11 -.01 -.03 .07 .03 .21** .09 Philosophical .16* .20** -.10 -.03 -.15* .08 -.06 .14* .10 Practical -.04 .04 -.18* .11 -.08 .15* .13 .03 .16* Quiet -.05 -.27** -.30** .08 .08 -.14* -.01 .03 .01 Relaxed .00 -.08 -.07 .30** .05 .07 .25** -.04 .07 Rude .06 .05 -.03 -.18* -.02 -.03 -.15* -.05 -.04 Shy -.10 -.30** -.12 .00 .15* -.21** -.06 .01 -.02 Sloppy .11 -.03 -.12 -.02 .06 -.09 -.02 -.07 .01 Sympathetic -.16* .01 .12 .04 .06 -.03 .04 -.06 -.05 Systematic -.09 .05 -.14* -.05 -.12 .07 -.06 .15* .13 Talkative -.02 .22** .17* -.10 -.07 .08 -.03 -.19** -.12 Temperamental -.02 .06 .04 -.30** -.03 -.12 -.30** -.13 -.10 Touchy .02 .04 .07 -.23** .02 -.14 -.24** -.11 -.10 Unattractive -.02 .00 -.09 -.08 -.04 -.09 -.14* -.09 -.11 Uncreative -.26** -.20** .13 .01 .19** -.05 .08 -.06 -.12 Unenvious .03 .08 -.08 .19** -.07 .09 .14 .07 .11 Unintellectual -.10 -.21** .17* -.07 .18* -.14 -.04 -.23** -.19** Unsympathetic .07 .01 -.13 -.12 -.06 .01 -.09 .05 .08 Warm -.09 -.03 .12 .09 .08 .04 .13 -.01 -.05 Withdrawn -.03 -.25** -.08 -.05 .12 -.28** -.17* .08 .04 Note. N = 196; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.


119

Table B.18 Judgment Scale Correlations with Observer Ratings on Descriptive Phrases 1 2 3 4 5 6 7 8 9 Finds fault with others .12 .11 -.09 -.28** -.06 -.03 -.26** -.02 -.03 Does a thorough job -.12 .03 .03 .02 -.09 .10 .04 .13 .01 Is depressed/blue -.07 -.11 .10 -.28** .08 -.26** -.32** -.04 -.08 Is reserved -.11 -.28** -.19** .09 .15* -.19** .00 .03 -.06 Can be somewhat careless .11 -.01 .04 -.07 .07 -.10 -.08 -.16* -.02 Relaxed/handles stress well .12 .11 -.09 .33** -.17* .20** .23** -.03 .10 Full of energy .22** .23** -.05 -.02 -.22** .22** .01 .03 .08 Starts quarrels with others -.01 -.05 .01 -.24** .09 -.11 -.18* -.10 .02 Can be moody -.02 -.11 .04 -.24** .11 -.18* -.21** -.05 -.03 A reliable worker .05 .08 -.01 .05 -.19** .15* .02 .07 .05 Can be tense -.01 .09 .10 -.32** -.03 -.06 -.25** .04 -.06 Ingenious/deep thinker .08 .24** -.21** -.12 -.28** .13 -.14 .14 .18** Generates a lot of enthusiasm .07 .29** .13 -.11 -.21** .13 -.10 .00 .01 Has a forgiving nature -.09 -.09 .12 .15* .14* .02 .19** .04 .03 Physically attractive -.07 -.03 .22** .05 .03 .06 .10 .09 .03 Tends to be disorganized .13 .02 -.05 -.01 .06 -.08 -.02 -.17* -.10 Worries a lot -.05 .00 .00 -.27** .02 -.15* -.25** -.07 -.05 Has an active imagination .21** .24** -.11 -.17* -.21** .06 -.19** .08 .07 Tends to be quiet -.06 -.28** -.28** .08 .10 -.15* .00 .08 .01 Generally trusting -.11 -.07 .06 .14 .02 .03 .13 .10 .10 Tends to be lazy -.03 -.23** .07 -.10 .21** -.17* -.04 .02 .03 Gets nervous easily -.13 -.14 .13 -.28** .08 -.28** -.30** -.04 -.08 Emotionally stable/not easily upset .04 .02 -.04 .38** -.03 .16* .33** .04 -.02 Inventive .16* .24** -.13 -.07 -.23** .11 -.10 .11 .11 Has an assertive personality .11 .39** .12 -.09 -.27** .23** -.05 -.06 -.05 Original/comes up with new ideas .15* .22** -.13 -.06 -.25** .10 -.13 .15* .12 Can be cold and aloof .10 .03 -.07 -.12 -.10 -.04 -.17* -.08 -.06 Not good-looking -.03 .01 -.15* -.10 -.05 -.07 -.16* -.09 -.06 Perseveres until the task is finished -.15* .13 .01 .07 -.21** .09 -.01 .09 .08 Values artistic, aesthetic experiences .14* .12 -.07 -.06 -.09 .04 -.08 .03 .11 Sometimes shy/inhibited -.07 -.36** -.16* .02 .24** -.21** .00 .04 .02 Considerate and kind to almost everyone -.17* -.02 .03 .11 .03 -.01 .08 .01 .03 Does things efficiently -.12 .07 .04 .03 -.16* .14* .05 .20** .11 Remains calm in tense situations .08 .09 -.08 .24** -.13 .19** .22** .01 .04 Prefers routine work -.32** -.34** .07 -.08 .29** -.18* .01 .08 -.04 Helpful & unselfish with others -.15* -.05 .05 .18* .03 .03 .16* -.07 -.04 Outgoing/sociable .09 .35** .16* .04 -.20** .24** .08 -.12 -.05 Sometimes rude to others .13 .06 -.08 -.16* -.02 -.05 -.16* -.01 .01 Makes plans and follows through -.06 .15* -.01 .21** -.13 .13 .15* .08 .15* Likes to reflect/play with ideas .13 .12 -.22** -.06 -.14 .09 -.06 .11 .13 Has few artistic interests -.14 -.13 .11 .11 .08 .03 .15* -.06 -.07 Likes to cooperate with others -.19** -.07 .08 .19** .10 -.02 .18** .06 .00 Easily distracted .03 .05 .06 -.24** .05 -.13 -.21** -.18* -.07 Sophisticated in art, music, literature .08 .10 -.16* -.07 .00 -.02 -.07 .12 .15* Curious about many different things .22** .15* -.23** -.04 -.13 .06 -.06 .01 .05 Note. N = 196; 1 = Threat Avoidance vs. Reward Seeking; 2 = Tactical vs. Strategic Thinking; 3 = Data-Driven vs. Intuitive Decisions; 4 = Defensive vs. Cool-Headed; 5 = Denial vs. Acceptance; 6 = Superficial vs. Genuine Engagement; 7 = Openness to Feedback & Coaching; 8 = Verbal Information Processing; 9 = Numerical Information Processing; * Correlation is significant at .05 level; ** Correlation is significant at .01 level.

judgment tech emanual - hogan assessments

Documents