IT Workforce Credentialing Summit: Overview of Competency Modeling and Assessment

November 21, 2008Presenters: Cheryl Paullin, HumRRO

Susan Baker, ITAA

Creating a Credentialing System Identify target competencies

For proof of concept For later, full-scale implementation

Buy/build assessments for target competencies Degree of fit to target competency Quality of the assessment Ease of administration & scoring

Scalability Time & cost for examinees and administrators

Collect evidence to prove assessments work Proof of concept Full-scale

Create baskets of proven assessments credentialing system Motivate individuals/organizations to use assessments

IT sector stakeholder endorsement Educational sector buy-in Distribute information widely to prospective test providers and test-takers ROI analyses

What is a Competency?

A measurable pattern of knowledge, skills, abilities, behaviors, and other characteristics that an individual needs to perform work roles or occupational functions successfully (US Office of Personnel Management)

Broadly, general & technical General = cognitive and social capabilities (e.g., problem

solving, interpersonal skills) Technical = knowledge and skill requirements for a specific

occupation or job

What is a Competency? Can be defined at varying levels of

specificity

Communication

Oral Communication Written Communication Listening Skills Briefing Skills

Knowledge of Rules of Grammar, Punctuation, etc.

Knowledge of Technical

Writing Standards &

Formats

Ability to Tailor Message &

Style to Intended Audience

What is a Competency?

Competencies can build on each other

Basic Interpersonal Skills

Conflict ManagementWorking Globally Networking

Where are the gaps?

Competencies that the IT sector needs but job applicants don’t have (enough of)

Structure discussion around an existing IT Sector competency modelAvoid reinventing the wheelProvides a common language

Existing Competency Models IT-Focused

Competency Model for the Information Technology Industry (ITAA, funded by DOL/ETA)

States’ Career Clusters – Information Technology Information Technology Skill Standards (NWCET) ICT Skill Standards (National Skill Standards Board and

subsequent efforts) Work- and Worker-Oriented Profiles Available for IT Sector Jobs

in O*NET (US DOL) Digital & Technological Literacy initiatives (e.g., California

Emerging Technology Fund (CETF)) General

Partnership for 21st Century Skills National Research Council Focusing on Future Skill Demands

Competency Model for the IT Industry: Background

President’s High Growth Job Training Initiative ITAA selected due to

Diverse membership 40 regional associations throughout US

Model released in October 2008, but builds on model originally developed by NWCET

Focus groups conducted in 4 technology hubs Seattle, Boston, Minneapolis, Washington DC Participants included educators, public workforce professionals,

IT leaders and professionals, service providers to IT

ITAA & US DOL/ETAhttp://www.careeronestop.org

National Skills Standards

Digital / IT Literacy

ICT Skill Standards

O*NET Profiles for IT Sector Jobs

P21 Skills

ECDL/EUCIP

ITAA / Dept of Labor IT Competency Model

Potential Existing Assessments

ICCP Core & AdvancedECDL Core & Professional

Hardware/Application/System-specific Certs

ICCP CoreECDL CorePending P21 Assessments

Work Keys (ACT)Pending P21 Assessments

ICT LiteracyAssessment

ECDL Equal SkillsOECD PISA

Key Discussion Questions

Where are the competency gaps?

Which gaps are most important to address first? Lead to most negative consequences for the business Hardest to train after hire

Which part of the existing gap does it make sense for us to try to tackle in order to prove this approach is viable?"

Key Discussion Questions (cont.)

What competencies/experience would a job applicant need to possess for your organization to consider that person for an entry-level IT job, even if he/she does not have a college degree?

Key Discussion Questions

What barriers exist, from your perspective, of getting to an industry-wide credentialing system?

Backup Slides

Existing Assessments Yellow & Blue Levels of the ITAA Model:

Computing Technology Industry Association (CompTIA) Certifications offered in PC hardware, networking, servers, e-

Business, convergence, project management, training, Linux, security, digital home technology, document imaging and RFID

Institution for Certification of Computing Professionals (ICCP) 1. Associate Computing Professional (ACP-new entrants into IT

<2yrs)2. Certified Business Intelligence Professional (CBIP - Global)3. Certified Computing Professional (CCP- Global)4. Certified Data Management Professionals (CDMP - Global) 5. Ceritified IT Compliance Professional (CITCP) - Global 6 . Certified IT Consultant (CITC) - Global7. Information Systems Analyst (ISA)

8. Information Systems Professional (I.S.P. - Canada) European/International Computer Driving License

Basic IT skills, Intermediate (“advanced” IT skills) European Certification of Informatics Professionals: Core &

Professional

Existing Assessments (continued)

Level 3 of the ITAA model: ICCP CoreECDL CorePending P21 Assessments

Existing Assessments (cont.)

Bottom 2 Levels of ITAA model:OECD Programme of International Student

Assessment (PISA)Work Keys (ACT)

Fundamental Life Skills Fundamental Academic Skills

Pending P21 Assessments

Partnership for 21st Century Skills

“Technical Competency” Organizing Structure

IT Industry Wide Competencies (ITAA) Principles of Information Technology Information Management Networks & Mobility Software Development User & Customer Support Digital Media Compliance Security & Data Integrity

“Technical Competency” Organizing Structure Career Pathways (States’ Career Cluster

Initiative)Network Systems Information & Support Services Interactive MediaProgramming & Software Development

“Technical Competency” Organizing Structure ICT Skills Concentrations (NSSB)

Database Development & AdministrationDigital MediaNetwork DevicesNetwork InfrastructureProgrammingTechnical WritingWeb Development & Administration

“Technical Competency” Organizing Structure IT Skill Standards (NWECT)

Database Development & Administration Digital Media Enterprise Systems Analysis & Integration Network Design & Administration Programming/Software Engineering Technical Support Technical Writing Web Development & Administration

“Technical Competency” Organizing Structure Occupational Technical Skills and

Concentrations (National Center for Information & Communication Technology Literacy) Common Skills Technical Writing Programming Database Administration & Development Network Infrastructure Network Devices Digital Media Web Development & Administration

Application Development

Customer Relationship, Service and Project

Management

Infrastructure EngineeringDevelopment

Application Services

Core Technologies

User Services

System Administration

Help-Desk and Technical Support

Services

Network and Services

AdministrationOperations

ICT Work Clusters

Figure from Wilcox, D. (2007, July). Rebuilding the IT organization for convergence. Business Communications Review, 46-51.

Overview of Assessment Assessments vary in how directly and accurately they

measure target competencies Simulation of actual work activities Work sample Evidence of relevant prior experience (resume, interview

questions, training & experience checklist, work products) Multiple-choice exam Self-report questionnaire

Generally, higher levels of realism mean More costly to develop and update More time to develop More time to administer or narrower range of coverage

Overview of Assessment (cont.) Assessments vary in the type of competency they attempt to measure

Management competencies (assessment center, role play exercises, work simulations)

Technical competencies (multiple-choice knowledge tests, work samples) Workplace competencies (work simulations, situational judgment test,

problem-solving tests) Academic competencies (reading test, critical thinking test, math test) Personal effectiveness competencies (integrity test, personality test,

assessment center) A college degree can be viewed as an “assessment”

Knowledge in a particular subject area (assumed if the student earned passing grades)

“Other things” learned during college Persistence to complete a course of study (maybe) Working on team projects Time management skills

Buy/Build Assessments OPM tool contains information about how to

measure competencieshttp://apps.opm.gov/ADT/content.aspxTool does not (yet) recommend specific assessments

for specific competencies Many vendors provide assessments

Assessment name may or may not reflect what it actually measures

High-volume testing is their sweet spot

OPM Decision Tool: Selected Competency by Assessment Method Matrix

COMPETENCIESAccomplishment Records

Assessment Centers

Biographical Data (Biodata) Tests

Emotional Intelligence Tests

Integrity/Honesty Tests

Job Knowledge Tests

Customer Service Preferred Preferred Preferred Potential - -

Information Technology Architecture

Preferred Potential Potential - - Preferred

Integrity/Honesty Preferred Potential Preferred - Preferred -

Oral Communication Preferred Preferred Potential Potential - -

Competency by Method Matrix (Cont.)

COMPETENCIESPersonality Tests

Reference Checking

Situational Judgment Tests

Structured Interviews

Training and Experience (T & E) Evaluations

Work Samples and Simulations

Customer Service Preferred Potential Preferred Preferred Potential Preferred

Information Technology Architecture

- Preferred Potential Preferred Potential Preferred

Integrity/Honesty Preferred Preferred Preferred Potential Potential Potential

Oral Communication - Preferred - Preferred Potential Potential

Assessment Examples

Level 1: Personal Effectiveness Competencies Workforce Readiness Credentials: Personal Skills Reid Report (Integrity test) Hogan Personality Inventory Learning Styles Indicator

Level 2: Academic Competencies Workforce Readiness

Credentials: Foundational Skills Numerical Reasoning Reading Comprehension Critical Thinking Tests

Assessment Examples

Level 3: Workplace Competencies Time Management Practices Inventory Team-based simulation/role play exercise Evidence of successful performance in

the workplace on target competencies (interview questions)

Level 5: Industry-Sector Technical Competencies Certifications offered for specific

hardware/systems PMI© Certification

Level 4: Industry-Wide Competencies IT Certification for Computing

Professionals Core Exam (ICT Center) College degree in Computer

Engineering/Computer Programming Evidence of successful performance in

an IT-relevant job (T&E checklist)

Scoping Questions

How many employees does your organization hire directly out of college?With a 4-year degreeWith a 2-year technical degree

What percentage of your IT-related job openings require NO:Prior work experiencePost-high school training

OPM Decision Tool: Situational Considerations for Purchased Assessments

Assessment MethodTimeframe Required

Level of Expertise Required

Maximum Suggested Volume

Level of Resources Required

Accomplishment Records < 3 Months Low 50-100 Medium

Assessment Centers 3 - 9 Months Low <50 High

Biographical Data (Biodata) Tests

< 3 Months Low Over 100 Low

Emotional Intelligence Tests < 3 Months Low Over 100 Low

Integrity/Honesty Tests < 3 Months Low Over 100 Low

Job Knowledge Tests < 3 Months Low Over 100 Low

Personality Tests < 3 Months Low Over 100 Low

Reference Checking < 3 Months Low <50 Low

Situational Judgment Tests < 3 Months Low Over 100 Low

Structured Interviews < 3 Months Low <50 Medium

Training and Experience (T & E) Evaluations

< 3 Months Low Over 100 Low

Work Samples and Simulations 3 - 9 Months Low <50 High