working smarter: job performance metrics for the … smarter - job...working smarter: job...
TRANSCRIPT
Working Smarter: Job Performance Metrics For the Smart Grid
SANS SCADA Conference Presentation
• Transforming the electric power infrastructure
• Lack of a viable cybersecurity workforce
• Our Nation’s epic journey
• Grid modernization efforts require:
- Advanced, continually maturing cybersecurity capabilities
North American Power Grid: Vast and Ever Growing Cyber-Attack Surface
Smart Grid Cybersecurity Panel
• NBISE – National Board of Security Examiners • The National Board • Job Performance Panels
- Operational Security Testing (OST)
- Advanced Threat Response (ATR)
- Smart Grid Cybersecurity (SGC)
We identify and measure the proficiency, the performance,
and the potential of the cyber security workforce.
Purpose: The project contributes to the Department of
Energy’s efforts to develop a competency model and explore
assessment methods focused on the job responsibilities and
unique skill set of Smart Grid cybersecurity specialists.
Who: Those primarily responsible for operational security
functions for day-to-day operations, but not engineering and
architecture, in smart grid environments.
Outcome: A measurement model for assessing
knowledge, skills, and abilities in the areas of technical and
operational skills.
Smart Grid Cybersecurity Panel Mission
• Managing Partner – UtiliSec
• Specializes in Smart Grid security architecture design and penetration testing.
• Led the Smart Grid Security Architecture group in the creation of NIST Interagency Report 7628
• Currently plays key roles in the Advanced Security Acceleration Project for the Smart Grid (ASAP-SG), National Electric Sector Cybersecurity Organization Resources (NESCOR), and Smart Grid Interoperability Panel (SGIP).
• Taught courses in hacking techniques, forensics, networking, and intrusion detection for multiple universities, corporations, and security conferences. Currently an instructor for the SANS Institute. In addition to electric power industry conferences,
• Frequently presents at top security conferences such as Black Hat, DEFCON, OWASP, and AusCERT.
• Co-leads prominent open source projects including the Samurai Web Testing Framework, Middler, Yokoso!, and Laudanum.
• MBA in International Technology
• CISSP and SANS GIAC certified Incident Handler (GCIH), Intrusion Analyst (GCIA), and Web Application Penetration Tester (GWAPT).
Justin Searle, SGC Panel Chair
Scott King, SGC Panel Vice Chair
• Manager, Security Operations – Sempra Energy
• Since 2001 information security field supporting the Department of Defense and shore based Navy and Marine Corps.
• Multiple roles within the security community supporting federal government, DoD, state government, commercial companies, and most recently critical infrastructure.
• For the past three years, Mr. King has worked for the Sempra Energy Utilities family of companies as the Security Operations Manager.
• He and his team have enterprise operational security responsibility for the corporate IT infrastructure as well as the gas and electric controls networks.
• Certifications
(CISSP, HISP, IAM, IEM)
Subject Matter Experts and Advisory Group
Panel Member Representation
Service
Government
Industry
Research
Vendor
SGC Panel Productivity
Master Vignettes
82
109
165
13
Job Responsibilities
26 Job Roles
Operational or Incident
Response Scenarios
Process Steps
Job Tasks 546
Target Cybersecurity Job Roles
Three job roles became the focus, based on Workforce Development and Job Performance Model process:
Security Operations Specialist
Intrusion Analyst
Incident Response Specialist
SMART GRID CYBERSECURITY
JOB PERFORMANCE MODEL
JOB ANALYSIS QUESTIONNAIRE
How can you get involved?
www.sgcjaq.nbisesites.org
Example JAQ Survey Questions
Job Analysis Questionnaire - Logistics
Participation will be limited to 15 minutes Survey participants will have the option to take an additional 15-minute portion of the survey.
The survey can be taken with any browser or web-capable device.
Target date for release: February 1, 2012
Thank You and Questions?
www.sgcjaq.nbisesites.org
To participate in the Smart Grid Cybersecurity
Job Analysis Questionnaire, please visit:
Backup
What is a Vignette?
A collection of: • a critical incident title or description • when the incident occurs (frequency and/or action
sequence) • what happens during the incident (problem or situation) • who is involved (entities or roles) • where the incident might happen, now or in the future
(systems or setting)
Further definition of a vignette might include: • why it is important (severity or priority of response) • how the critical incident is addressed (method or tools that
might be used)
Elicitation Tools and Methods
Incident Response Vignette Identification & Elaboration
Elicitation Tools and Methods
Vignette Master Categories
• Data Leakage/THEFT [9075]
• Network Attacks [9076]
• Substation/SCADA Attacks [9077]
• AMI Attacks [9078]
• Client Side Attacks [9079]
• Phishing Incidents [9080]
• Network Separation and attack paths [9081]
• Incident Response Process & Log Management [9082]
• Encryption Attacks [9083]
• Security Testing [9084]
• Threat & Vulnerability Management [9085]
• Access Control Maintenance [9086]
• Risk management, compliance and audit [9087]
Elicitation Tools and Methods
Three key roles based on SGC Panel elicitation exercises
The SGC panel selected three job roles based on dozens of candidate roles:
Elicitation Tools and Methods
SGC Job Responsibilities & Required Job Tasks
Elicitation Tools and Methods
(Example responsibility with its tasks)
Responsibilities & Task Mapping
Elicitation Tools and Methods
Task Identification & Voting
Elicitation Tools and Methods
In Summary
• Support the development of this Job Classification Report and job roles
• Ready for broader industry contribution
• Establish the context for an accelerated training
program
• Advance a measurement model
• Validate the performance model
• Create foundation for assessing performance during
and after training
Results, Innovations, Future Plans
• Context elicitation through vignettes and responsibilities
• Competencies defined at novice, apprentice, expert and
master levels for multiple roles (organizational language)
• Creating profiles for technical and operational skills
• Reusable libraries to assess multiple roles
• Validation and extension of job performance model Standards based on:
- Validated curricula
- Assessment
- Simulation libraries
How is the NBISE Approaching Competency Different?
SGC Panel - The difference
Put KSA diag here
• The Job Analysis Questionnaire (JAQ), is broadly disseminated within the Electric Sector community.
• The JAQ seeks to gauge both the relative importance of given tasks and
the frequency with which they are conducted.
• In parallel with survey data analysis, NBISE staff and researchers work
with experts to augment the growing competency model report with critical incident analysis.
• Intensive, in-depth interviews are geared toward documenting
characteristic and challenging events that embody the most important aspects of the job.
• Further work is done with these “critical incidents” and “situational
judgment scenarios” to determine how the actions taken by a novice, apprentice, journeyman, and expert-level practitioner are differentiated.
The SGC Job Analysis Questionnaire encourages broad workforce input
SGC JAQ Survey Initiative
Competency Modeling Process
SGC Panel - Phase 1 Process