C E E RCyber-Physical Testbeds

(a generational leap)

CEER: Cyber-Physical Experimentation

TESTBED

ASSETS

PEOPLE

SCIENCE

DATA

PROVISIONCustomer

CLOUD

LOCAL

(testbed operation support)

OtherTestbeds

TestbedLineageAnEvolution

InceptionIdentifiedneedsandstartedonsolutionforNSFTCIPproject

2005 2008 2010 2016

InternaltoExternalBegantransitiontosupportexternalexternalusers,viacollaborativetoolsandfederation

EvolutionandGrowthEvolvedcapabilitiesandincreasedcapacity(largelyforDOEsupportedprojects)

RefineandExpandRefineusabilityandexpandcapabilities,changingthetestbedlandscape

FutureExpandcapabilitiesinothercriticalinfrastructuredomains

FUTURE

Usability

Value proposition

Scientific Advances

Increase the usability, accessibility, and capability of evaluation / experimentation infrastructure

Continue to tackle needed capabilities that cannot be provided in facilities today

Advance the state of art to provide support for research and development of next generation OT forensics and cyber-system evaluation tools

Capability

Our approach

ModularityAdaptable

composition, configuration, and

deployment of testbed assets to

accomplish exercise goals

InstrumentationAppropriate and

accurate instrumentation to capture needed

assessment knowledge without

affecting results

KnowledgeBlend of academic,

enterprise, manufacturing, and

asset owner knowledge to ensure a multi-dimensional

approach

Driven ModelsScalable, accurate, and encompassing cyber and physical

models that adapt to exercise needs

based on performer input

10,00 Mile View

• Physical testbed access• Dedicated (isolated) office

space on site• Dedicated remote access• ICS software and

equipment• Computation and Storage

support (within reason)• Capacity to bring in special

software and equipment

Capabilities• Full end-to-end Smart Grid capabilities• Deployed Advanced Metering Infrastructure (AMI)• Solar research platforms• Real, emulated, and simulated hardware/software for scalability• Real data from the grid, Industry partners, etc.• Power simulation, modeling, and optimization of various forms• Network simulation, modeling, and visualization of various forms• Advanced hardware-in-the-loop cyber-physical simulation• WAN/LAN/HAN integration and probes• Security and protocol assessment tools (static/dynamic analysis, test harnesses,

fuzzing)• On-grid testing capabilities via Ameren TAC facility (with fiber optic interconnects

to our primary testbed)

System Details (current)Firewall

Switch C1

TechServicesManaged Router

Illinois Campus Network Core

Switch C2 Switch C3

Compute Cluster

TestbedFloor/Wall

Jacks

Switch R2

RTDS Racks

SDNTestbed

Armore

EMS

OSIsoftPI System

AmerenTAC

Switch R1

TriconexDigital Nuclear

Reactor Controller

BWRReactor Simulator

Switch R3

RouterTestbed

Switch R5Switch R4

GPS Clocks GPS Clocks

Dual ConstellationGPS Antenna

Dual ConstellationGPS Antenna

Roof

Substation Computers

SEL Security GatewaySEL Security Gateway

SEL Relays

SEL Relays

GE Relay

GE Relay

ABB RelaysArbiter PMU

ABB Relays

OrionLX

SEL AMSSEL AMS

SEL RTAC SEL Relays

10G Fiber<vlan 822>

10G Copper<vlan 822>

10G Copper<vlan 1>

10G Copper802.1q

10 Gbs FiberUI TechServices10 Gbs CopperTestbed Core1 Gbs Copper

Testbed Peripheral

10 Gbs VRFUI TechServices

10 Gbs CopperFirewall

Challenges

• “It’s Impossible”• Duplication of effort on the same problem• “Safety” (protection of national infrastructure) vs Realism• Automating both cyber and physical components• Lowering the barrier to entry broadly• Need open datasets that are more than just synthetic• Seamless operation regardless of locale• Integration of black boxes• Existing tools aren’t very good

Observations

• Universal authentication and access scheduling is not a solved problem• Fully dynamic asset management bucks ICS industry trends• Existing automation/cyber experimentation frameworks are not

sufficient• Co-Simulation systems work… but are not very flexible• ICS datasets and tools to generate them are sparce• Federation is helpful, but only in very specific use cases due to

unsolved science problems• Usability, flexibility and accessibility are key

Salient Points

• Be Open!• How-to, data, tools, etc.

• Focusing on infrastructure/assets alone is short-sighted• Scientific advances are necessary in our space• “Field of Dreams” doesn’t really apply• Breaking down barriers is absolutely required• We need systems that enable not dissuade• We need communities to promote, discuss, and exchange knowledge• We need to be open and work together to realize the true potential• Stop just competing! Collaborate instead.

Testbed Donations Provided By

THANK YOUTimYardley

AssociateDirectorofTechnology

UniversityofIllinoisUrbana-Champaign

yardley@illinois.edu