Accelerator Control-System Network Security@Diamond Light Source

Mike Leech, Controls Group Computer Systems Manager

Dream Accelerator Controls Network?++Isolated+No routing, Layer 2 only Easy configuration and hardware replacement+Simple Star network no daisy-chaining+ Cheap

-No diverse routes for fibres-No automatic hardware failover

Primary Network:All EPICS control trafficOnly primary network needed to run machine. All services contained within DNS, NFS, NTP, IOC boot (FTP), Parameter archiving etc.

Devices on the Primary Network:16 Linux Servers11 Linux CA Gateways45 Linux Workstations298 VxWorks IOCs222 Linux BPMs4 Windows PCs40 Linux/Windows Laptops6 Other!!! (Atomic Clock, GPIB adapter, etc)0 PLCs!!! (All PLCs hang off private networks on IOC second interface)

Secondary Network:All non EPICS traffic and traffic not essential to machine operationVideo cameras, scopes, terminal servers, IP phones, pump carts, residual gas analysers, printers etc.Nearly identical to primary network except, routed to allow access to dual homed servers and workstations.

Powerful security tools out-of-the-box: Iptables stateful firewall, tcpwrappers (hosts.allow), SSH encrypted login shell (copying, tunnelling and more).Open Source: Security flaws discovered and patched quickly.Secure services: VSFTP, Apache, SELinux Jail.Total control over system configuration rebuild your own kernel.Security through obscurity: Less of a target for viruses and worms.No Power Users unless you configure elevated rights

Dual Homed Servers:SSH Bastion: Allows remote access during shutdown and emergency remote access during operation to fix faultsEPICS Channel Access archiver: Allows office access to archived data.Bootserver: Allows office read-only access to software (3.14).Relational Database: Allows access to ELog, cable schedules etc

Diamond Control Room

Physical Access:Network access points are restricted to the following locations:Control and instrumentation areas (CIAs).Linac, booster and storage ring tunnels.Computer room.Control room.Comms rooms.NO labs or offices.NO wireless.All these areas are under access control.

Bridging (Stealth) firewall:Close down both interfaces:> ifdown eth0; ifdown eth1 > ifconfig eth0 0.0.0.0 > ifconfig eth1 0.0.0.0 Create a bridge:> brctl addbr br0 Add both interfaces:> brctl addif br0 eth0 > brctl addif br0 eth1 Turn on IP forwarding:> echo 1 > /proc/sys/net/ipv4/ip_forward Configure management interface:> ifconfig br0 172.23.0.1 netmask 255.255.255.0 up

Iptables firewall:> iptables -F > iptables P FORWARD DROP > iptables P INPUT DROP; IPTABLES P OUTPUT DROP > iptables A INPUT i lo ACCEPT; iptables A OUTPUT o lo ACCEPTRestrict by interface:> iptables A FORWARD i eth1 o eth0 p tcp --dport 22 j ACCEPTRestrict by IP address range:> iptables -A FORWARD --destination 172.23.0.0/16 -p udp --dport 53 -j ACCEPTStateful:> iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPTeth1eth0

Epics Channel Access Gateways:Enable machine parameters to be read from isolated primary networkOne for office networks and one for each beamline networkApplication layer gateways. No direct routing of IP packetsUnidirectional read-only gateway for officeBidirectional read-only gateway per beamline no default routeCA monitor allows moving of ID gaps through read only gateway

Diamonds Public and Private Networks:Diamonds control, office, science and beamline networks are all NATd private networksSome proxyed protocols eg. Real player, http, httpsA limited number of other protocols allowed out eg. sshDiamond controls public network has a public address range and is directly routed to diamond private networks, but behind site firewallSSH bastion and reverse web proxy on public networkNo DMZ - yet!.

Apache Reverse Web Proxy:Enables one web server to provide content from another transparently.Gives encrypted and authenticated access to certain internal web pages. Such as, Elog, archiver, Machine status.http://internal.com -> https://external.com/internal

LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule headers_module modules/mod_headers.so LoadFile /usr/lib/libxml2.so LoadModule proxy_html_module modules/mod_proxy_html.so

DocumentRoot /var/www/html/external ServerName external.com

ProxyPass /internal/ http://www.internal.com/ ProxyPassReverse / SetOutputFilter proxy-html ProxyHTMLURLMap / /internal/ ProxyHTMLURLMap /internal /internal RequestHeader unset Accept-Encoding

SSLProxyEngine on AuthType Basic AuthName External Area" require valid-user Allow from all

SSLEngine on SSLCertificateFile /etc/httpd/conf/ssl.crt/external.crt SSLCertificateKeyFile /etc/httpd/conf/ssl.key/external.key

AcknowledgementsI would like to thank the following for their help:

Tim HaytonOriginal Network Design and TenderMark HeronOriginal Network Design / Contract ManagerPete LeicesterChannel Access GatewaysPeter Denison, James RowlandLINUX / EPICS GurusFrederik Ferner, Tina FriedrichMore LINUX GurusPaul Amos, Simon LayOriginal Cable InstallationChris Colbourne, Nico RotoloCabling MaintenanceThe LINUX and EPICS communities

Network security may seem like an impossible struggle!But dont give up hope ;-)

Diamond is a new third generation light source near Oxford in the UK.First Light May last year. First users January of this year.Im the computer systems manager for the controls group, but for my sins Ive also inherited the running of the controls network.Ill be talking only about the accelerator control networks not science or beamline networks which Im not responsible for.When we set out to design diamonds controls network structure we set down two main criteria.The network had to be both stable and secure.The first design decision was then simple the network should be isolated.Then we took a look at the most common causes of network downtime on site: Rouge devices. Incorrect configuration: VLANs, routing tables, redundant links, spanning tree. Hardware failure. Cut / broken fibres.A combination of rouge devices and complex networks can lead to very difficult to diagnose problems!We took our second decision and kept the network as simple as possible: No routing and no VLANS One flat layer 2 network. With a large core switch and directly attached edge switches in each instrumentation area (CIA). Simple star network, no stacked switches, no daisy chained switches, no aggregators very easy to diagnose problems.Gigabit fibre core 3com 7700, 100Mbit to IOCs 3com 3200s.Very cheap.

Downsides:No diverse routes for fibres we do have a timing network with dark fibre which can be used in an emergency.No automatic failover of hardware however hardware spares can be used out-of-the-box because we have such a simple layer 2 configuration.

We got our isolated network. We call it the primary network, and its the only network we need to control the machine.All the accelerator IOCs, servers and services, and workstations sit on the primary network.So far so good.

We got our isolated network. We call it the primary network, and its the only network we need to control the machine.All the accelerator IOCs, servers and services, and workstations sit on the primary network.So far so good.

Unfortunately we have a lot of other equipment that, although isnt directly necessary for running the accelerator, is still required from time to time, such as:Scopes, pump carts, residual gas analysers, video cameras, terminal servers etc.We dont want the network traffic from these devices interfering with our EPICS traffic, so we placed them on a separate network which we call the secondary network.

The secondary network is almost identical to the primary network, same core and edge switches, same fibre backbone, etc.The only real difference is that the secondary network is routed to the other diamond networks to allow access to the dual-homed servers, which Ill talk about later.

All operations servers and workstations at diamond run LINUX OS. Namely RedHat Enterprise Linux 4.We think Linux gives us a more secure platform out-of-the-box than foe instance, WindowsXP.It comes with powerful security tools: Iptables stateful firewall. Easy syntax, powerful features such as NAT, connection tracking modules eg. For FTP. SSH. Encryped login shell, but also much more: secure copying of files, secure FTP, tunneling. Secure services like very secure FTP VSFTP, Apache web server.Which can be jailed from the rest of the OS with security enhances linux SELinux, so if that service is compromised, it can limit the amount of harm can do to the rest of the system.(SELinux can give services minimum amount of privileges required to do there job, for instance will not we allowed to execute programs outside of its profile even if the file program is set to be executable by anyone. Created by the NSA. Part of kernel)With linux we have total control over the system. We can rebuild the kernel to remove unwanted features and add security features, of which SELinux is a good example. (Support for NX Bit).I dont like the term security through obscurity, because Linux is no longer an obscure OS. But it is true to say that LINUX is not nearly as big a target as Windows for Worms and Viruses.Due to its server roots, LINUX users have far fewer privileges than a normal windows user. For instance who owns the CDROM device and COM port. These rights must be assigned in LINUX through various tools like sudoAll well and good having an isolated control network, but how does someone fix a problem from home at 3am with no way to access the primary network.And operators want to be able to see the electronic log entries from their office.Software developers will want to be able to update the software on the boot server.Someone might want to look at archived machine parameters to detect trends.This is where we have to compromise on security slightly to allow for usability.At Diamond we have a few dual-homed servers with on interface on the primary network and one firewalled interface on the secondary network.See slide.How do other people allow for remote access. Currently ops can run a script to open and close the firewall. One-time-passwords. Could keep hold of them time expired?

Unfortunately, Ive been brow-beaten into dual-homing to the control consoles.They are firewalled to not allow incoming connections on their secondary network interface. But even so, trogans could find there way onto the machines that make outgoing connections. Im going to look into the possibility of restricting these consoles further, or even removing their secondary network interface altogether and using secure proxies to allow limited access to the outside world.

At Diamond we use dedicated hardware for routing, but this hardware has very limited firewalling capability.Dedicated firewalls expensive, learn new syntax and require expensive maintenance contracts and spares. Linux stealth firewall runs on a standard server platform with free software, so is cheap, and can use existing knowledge base of iptables syntax for easy maintainance.Network traffic coming in on any of these ports will be forwarded to the other ports transparently, so that the bridge is invisible to the rest of the network. Basically acts as a switch.Can add more interfaces, spanning tree, proxy_arp.Potentially very useful if you want to transparently firewall low security devices such as VXWorks IOCs and PLCs which dont have their own firewalling software.

At Diamond we use dedicated hardware for routing, but this hardware has very limited firewalling capability.Dedicated firewalls expensive, learn new syntax and require expensive maintenance contracts and spares. Linux stealth firewall runs on a standard server platform with free software, so is cheap, and can use existing knowledge base of iptables syntax for easy maintainance.Network traffic coming in on any of these ports will be forwarded to the other ports transparently, so that the bridge is invisible to the rest of the network. Basically acts as a switch.Can add more interfaces, spanning tree, proxy_arp.Potentially very useful if you want to transparently firewall low security devices such as VXWorks IOCs and PLCs which dont have their own firewalling software.

All well and good having an isolated control network, but how does someone fix a problem from home at 3am with no way to access the primary network.And operators want to be able to see the electronic log entries from their office.Software developers will want to be able to update the software on the boot server.Someone might want to look at archived machine parameters to detect trends.This is where we have to compromise on security slightly to allow for usability.At Diamond we have a few dual-homed servers with on interface on the primary network and one firewalled interface on the secondary network.See slide.How do other people allow for remote access. Currently ops can run a script to open and close the firewall. One-time-passwords. Could keep hold of them time expired?

All well and good having an isolated control network, but how does someone fix a problem from home at 3am with no way to access the primary network.And operators want to be able to see the electronic log entries from their office.Software developers will want to be able to update the software on the boot server.Someone might want to look at archived machine parameters to detect trends.This is where we have to compromise on security slightly to allow for usability.At Diamond we have a few dual-homed servers with on interface on the primary network and one firewalled interface on the secondary network.See slide.How do other people allow for remote access. Currently ops can run a script to open and close the firewall. One-time-passwords. Could keep hold of them time expired?