tyson jarrett cip enforcement analyst practices for security patch... · tyson jarrett cip...

Tyson JarrettCIP Enforcement Analyst

Best Practices for Security Patch ManagementOctober 24, 2013

Anaheim, CA

2

• Graduated from the University of Utah with a Masters in Information Systems

• Have been with WECC for 2 years 11 months

• Reviewed 1,407 CIP items• Ran my first Marathon this Month

A little about me…

Wrong reasons to Patch

4

• Cyber Security Patches are key to avert many known vulnerabilities to Cyber Assets and the environment

• Identified by ICS – Cert as one of the top security challenges within Industrial Control Systems

• Most importantly…

Why Patch Management

Requested by YOU!!

5

• The intent of R3 is to know, track, and mitigate known software vulnerabilities associated with BPS Cyber Assets. It is notintended as an “install every security patch” requirement; instead it should be considered more of a “be aware of in a timely manner, and manage all known vulnerabilities” requirement.

Intent of CIP 007 R3

6

• Security Patch Management — The Responsible Entity, either separately or as a component of the documented configuration management process specified in CIP-003-3 Requirement R6, shall establish, document and implement a security patch management program for tracking, evaluating, testing, and installing applicable cyber security software patches for all Cyber Assets within the Electronic Security Perimeter(s). o R3.1. The Responsible Entity shall document the assessment of

security patches and security upgrades for applicability within thirty calendar days of availability of the patches or upgrades.

o R3.2. The Responsible Entity shall document the implementation of security patches. In any case where the patch is not installed, the Responsible Entity shall document compensating measure(s) applied to mitigate risk exposure.

CIP 007-3 R3

7

• Tracking, Evaluating, Testing, and Installing applicable cyber security software patcheso Common Pitfallso Best Practiceso Audit/Enforcement Approach

Agenda

Tracking

9

Only tracking patches available at the Operating System level

• Entity only tracks patches with Windows Server Update Service (WSUS)o WSUS does not actively identify or track other

applications and/or software running on the Windows box

o Thus all third-party applications on the device are not being actively tracked

Tracking – Common Pitfall #1

10

Not maintaining appropriate documentation• Including:

o Incomplete or inaccurate list of devices and applications running on those devices

o Not knowing or documenting where patch releases are located.

Leads to systems not getting patched for known security vulnerabilities

Tracking – Common Pitfall #2

Tracking – Best Practices

Patch and Upgrade source

identification

Patch and Upgrade

identification

Asset identification

Patch andUpgrade source

identification

ide

Patch Tracking Process

12

• Asset Identificationo Ensure all applicable cyber assets

are documentedLeverage other CIP requirements (CIP 002 R3 and CIP 005 R1.6) when identifying assets.Include step to periodically verify accuracy of list

o Use combination of automated tools and manual walkthroughs/ verifications to ensure list is accurate



identification

Patch and

Upgrade identificat

ion

Paaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaatttttttttttttttttttttttttttcch aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaannnndddd

ppppppppppppppppppppppppppppppppppppppppppppppppppppppggggggggggggggggggggggggggrade ennnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnntttttttttttttiiffiiccaatt

Asset identification

Upgrade source

identification

ide

PaPPPPPPaPPPPPPPPPPPPaPaPaPPaPaPPaPatttttttctttctctcttctctctctctcccctctcccctctctctctctctcchhhhhhhhhhhhhhhhhhh aaanananannanaaaaaanaaaaaananananananddddddddddddddddddddddddddddddddddd

ioooooooooooooooooooooooonnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

13

• Patch/Upgrade Identificationo Identify and document all

applications, Operating Systems, and firmware on cyber assets

Minimize applications on CIP 007 applicable devices to only what is necessaryInclude steps to periodically verify accuracy of list



identification

Asset Identifica

tion

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAssssssssssssssssseeeetttt eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeennnnntificattttttttttttttttttttttttttttttttttttttttttiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiooooooooooooooooooooooooonn

Patch and Upgrade

identificationsUpgrade source

identification

PaPPPPPPaPPPPPPPPPPPPaPaPaPPaPaPPaPatttttttctttctctcttctctctctctcccctctcccctctctctctctctchhhhhhhhhhhhhhhhhhh aaanananannanaaaaaanaaaaananananananddddddddddddddddddddddddddddddddddd

tiiiioooo

14

• Patch/Upgrade Source identification and notificationo Where are the patches located?o How will you get notified of a new patch?

Vendor emailManually visiting webpageAutomated scanner (WSUS, patch management software)

o Implement periodic manual checks to verify automated solutions are functioning properly


Patch and Upgrade

identification

Asset Identific

ation

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAsssssssssssssssssssssssssssssssssssssssssssssssssseett eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeentificaaaattttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttiiiiiiiiiiiiioooooooonn


identification

Patch andUpgrade

identification

a

Patch and

atttttttttttttttttttttiiiiiiiooo nonid

15

• Patch Management toolso Commercial Softwareo Databaseo Spreadsheeto Papero Brain


Don’t do this!!Don’t do this!!


ProsBuilt with the intent to track and manage patches and upgrades

Vendor support can reduce need for in-house expertise

Can come with useful features• Asset identification, automated

notifications, baseline creation and update, vulnerability scanning

ConsEvidence presentation and retention may need additional planning

Research needs to be done ahead of time to ensure the right product is chosen• See SANS – “A Practical Methodology for

Implementing a Patch management Process” for 9 items to consider when picking an automated solution. (PG. 7)

•Commercial Vendor

17

• Some Commercial Vendors



Pros

Reduces redundancy

Reporting features allow for evidence retrieval

Can reduce data entry, storage, and retrieval costs

Cons

Can be complex and difficult to implement

Not always practical to build from scratch

May need new personnel familiar with creating and maintaining a database• Personnel may need additional

training

•Database


•SpreadsheetPros

Easy to implement

Low cost

ConsUpdating data can be difficult and often requires creating a new spreadsheet to maintain historical evidence

Can require repetition with other processes for updating data• Current patch version, work orders, etc

Useful information usually needs to be stored on another spreadsheet.• What devices have what applications/OS/firmware• Where patches are located• How are notifications being received

20

• Maintain documented procedures for tracking patches and updates

• Evidence of actively monitoring all available software and firmwareo Develop a list of all monitored applications/OS/firmware o Identify and document process and location for

notifications of updateso All applications/Operating Systems/firmware that

MAY receive security patches must be accounted for in Patch Management tracking procedures.

0

Tracking – Audit Approach

21

Evaluating

22

• Relying on Industrial Control System (ICS) vendor to evaluate applicability of patcheso Due to fear of voiding warranty with ICS vendor

entity leaves all patch management responsibilities to the ICS vendor

o Entity does not have procedures or timeline in place for evaluating patch applicability

Evaluating – Common Pitfall #1

23

• Not consistently evaluating patches within 30 days of availabilityo Entity tracks patches once a month

Thus entity continuously misses 30 day deadline as it does not have enough time to evaluate patches

o SMEs mis-read documentation and didn’t verify if all software had patches available

Evaluating – Common Pitfall #2

24

• Identifyo How assessment will be

documentedo Specific personnel

responsible for assessing the patches and upgrades

Should have collaboration with both IT and operations staff

4

Evaluating – Best Practice

25

• Implement a peer review process to verify evaluations are done correctly and necessary documentation is maintained

• Conduct periodic training on evaluation procedures and required documentation


26

• Plan aheado Track patches at least every two weeks to

ensure enough time is available to evaluate patches within the required 30 days of availability


27

• Don’t rely on ICS vendor to evaluate patcheso Determining a patch is applicable will not void

a warrantyEntity’s may still elect to wait for ICS vendor approval prior to installing a patch

o ICS vendors are not required by CIP to assess patches immediately, YOU are!


28

• Documented process for determining if patches and upgrades are applicableo An assessment should consist of determination

of the applicability of each patch to the entity’s specific environment and systems. Applicability determination is based primarily on whether the patch applies to a specific software or hardware component that the entity does have installed in an applicable Cyber Asset.

Evaluating – Audit Approach

29

• Must maintain evidence of identification and evaluation of applicability within 30 days of availabilityo Evidence should include date patch or upgrade

was made available, date it was evaluated, and evidence the documented evaluation process was followed

Evaluating – Audit Approach

30

Testing

31

• Patches are automatically installed, and thus not tested prior to installationo Entity was unaware

device was configured for automated updating

Testing – Common Pitfall #1

32

• Conducted functional testing onlyo Entity’s testing procedures required security

patches be installed in test system for 1 week then deployed to production if nothing broke

• Entity does not identify or document security controls impacted by the patch being installed

Testing – Common Pitfall #2

33

• Use existing CIP 007-3 R1 Test Procedureso Don’t re-invent the wheel!!

• Specific Documentationo Identify what tests are performed when and why?o Who is responsible for conducting the testing?

Who is responsible for approving the test?o What do the results of the testing mean?

What is a pass or fail?

Testing – Best Practices

34

• Implement peer review process to verify testing was done per procedures

• Implement periodic follow-up testing to validate current testing procedures are capturing data needed to make installation decision

Testing – Best Practices

35

• Disable automatic updateso Configure software to

notify but not installo Implement verification

process to periodically check for any cases where patches aren’t being tested

Testing – Best Practice

36

• At minimum must be compliant with CIP 007 R1 testing procedureso From CIP 007-3 R1 “a significant change shall,

at minimum, include implementation of security patches”

o Technical narrative describing testing environment(s)

Include how is test environment similar/ dissimilar to production environment

Testing – Audit Approach

37

• Documented testing procedures for each cyber asset (or asset type) within the ESP

• At minimum testing needs to ensure existing security controls are not adversely affectedo Before and after the test identify and document ports

and services, user accounts, Logging/Alerting functionality, and anti-virus.

• Controls should be in place to protect production environmento Separate test environmento Back out plan

Testing – Audit Approach

Installing

39

Installing - Common Pitfalls

• Not updating baseline after the change is madeo Personnel were unaware who was

supposed to update the baseline documentation

o Procedures didn’t explicitly call out updating documentation

40

• Leverage CIP 003 R6 Change Control and Configuration Management procedureso When installing a security patcho Following procedures during installo Updating baseline after the change

• Identify who is responsible for installing the patch or upgrade and updating documentation afterwards

Installing – Best Practice

41

• Use checklists to help SMEs easily identify all that is required as part of the installation

• Procedures should identify an acceptable time frame to install an applicable patcho Note: Requirement does not specify a required

time to install a patch or upgrade• Create a “back-out” plan

o Ensure backups and recovery plan are up to date and tested

Installing – Best Practice

42

• Documented procedures for installing security patches and upgradeso Evidence of installation of patches as defined in

documented procedures

• Evidence Documentationo Cyber Assets patch/upgrade was installed on

Who installed the patcho Updated device baseline after installationo Date of installation

Installing - Audit Approach

43

• If patch or upgrade is applicable but not installed, must implement and document Compensating Measureso Additionally a Technical Feasibility

Exception (TFE) may need to be submitted if the device will not be installing any new patches.

Installing – Audit Approach

44

• Procedures must include methods for Tracking, Evaluating, Testing, and Installing security patches

• Take extra time planning how patches will be tracked and documented to lessen burden of patch management further down the road

• Entity’s are responsible for compliance, not the entity’s ICS vendor

Summary

45

• Ben Christensen's “Root Cause Analysis for Commonly Violated Requirements”

• Sans “Practical Methodology for Implementing a Patch management Process”

• ICS – CERT “Improving Inductrial Control Systems Cybersecurity with Defense-In-Depth Strategies”

References

Tyson JarrettCIP Enforcement [email protected]

Questions?

tyson jarrett cip enforcement analyst practices for security patch... · tyson jarrett cip...

Documents