robust overlay networks for microgrid control systems
TRANSCRIPT
Robust Overlay Networks for Microgrid Control Systems
Geert Deconinck1, Tom Rigole1, Hakem Beitollahi1, Rui Duan1, Bart Nauwelaers1, Emmanuel Van Lil1, Johan Driesen1, Ronnie Belmans1, Giovanna Dondossola2
1K.U.Leuven – ESAT/ELECTA, Belgium. 2CESI RICERCA SpA, Grid and Infrastructures Department, Italy
(corresponding author: [email protected])
Abstract
Control systems for electrical microgrids rely ever more on an information infrastructure, consisting of heterogeneous off-the-shelf communication technology for hardware, software and networking. This provides flexibility for the application as new services become possible, but also implies vulnerabilities as the correct execution of microgrid control algorithms can be hampered by the incorrect functioning of the information infrastructure, as a result of accidental and malicious faults. In order to provide resilient microgrid control, the info’structure needs to be fault-tolerant and able to deal with a dynamic environment; overlay networks can provide the required graceful degradation in case of unrecovered failures, rather than resulting in a complete breakdown.
1 Introduction
The number of Dispersed Energy Resources (DERs) in electricity networks is constantly increasing (e.g. wind turbines, photovoltaics, coupled heat-power). This requires distributed control and hence distributed computer and communication systems. Main factors causing this shift from centralised electricity production to a decentralised power generation are the availability of small-scale units, which offer an increased flexibility in the liberalized energy market, and the growing trend towards sustainable development which favours energy efficient and low CO2 emitting plants [1, 2].
As more and more small scale dispersed generators are being deployed in the distribution grid, this puts extra stress on the power grid in an era where electricity is one of the most important commodities for economical, industrial and everyday activities. Regarding power system reliability, line losses, voltage profiles, etc., integrating DER units can bring benefits,
as well as deteriorated grid performance. Therefore new control strategies are being proposed to maintain the desired degree of dependability for electricity supply [3-5]. Many of these control algorithms are distributed over several computing nodes and rely on off-the-shelf ICT infrastructure for communication. Many DERs and loads become intelligent electronic devices (IEDs) that are interconnected over this information infrastructure (info’structure), as indicated in Figure 1. Also, new services can be delivered by exploiting both the power and info’structure simultaneously. For instance, external information, such as the instantaneous electricity price from real-time market places, can be incorporated into the control strategies in order to optimise economic benefits; intelligent loads can be switched on or off in order to implement demand side management and avoid costly electricity peak costs; etc.
electricity grid
Figure 1: Microgrids, consisting of DER
interconnected via the electricity grid (thick lines) and corresponding IEDs interconnected
via the info'structure (dashed lines). Dispersed electricity generation is proliferating
rapidly; this requires an equally proliferating, dependable, ICT infrastructure to support this. If sufficient generation (and storage) facilities are
info’structure
IED+DER
available in a part of the electrical grid, such part can become an energy island (or microgrid) which functions independently from the major grid (e.g. during a blackout or for economic reasons). However, in such islanding mode, control is different from non-islanding mode. Several technical issues need to be solved regarding protection and control by power engineers. However, many of the proposed solutions require an appropriate communication and control infrastructure that continues to function in both modes. It is indispensable that this info’structure is dependable.
This info’structure follows the trend of deploying heterogeneous off-the-shelf information and communication technology for hardware, software and networking [6]. This provides application flexibility, but implies vulnerabilities as the electrical energy infrastructure depends on the correct functioning of the info’structure in spite of accidental and malicious faults.
Typical examples of malicious faults to the info’structure of power systems include [7]: • Denial of Service (DoS) attacks on control systems
via telecom backbone. • Intrusions into Centre-Substation communication
flow and execution of faked commands (spoofing, man-in-the-middle attacks).
• Exploiting vulnerabilities in standardised application layer protocols.
• Accidental or malicious infections by worms or viruses in the substation network caused by maintenance or not-allowed activities of control personnel.
• Intrusions and viruses through ICT devices for the Primary, Secondary and Tertiary voltage and frequency control of generation power plants. In order to provide robust behaviour for energy
applications, this information infrastructure needs to be fault-tolerant and be able to deal with a dynamic environment; middleware can provide the required graceful degradation in case of unrecovered failures, rather than resulting in a complete breakdown [6, 8, 9].
Table 1: microgrid control applications non real-time real-time local data aggregation,
logging primary control (droop control)
distributed smart metering, systemmonitoring, demand side management, peakshaving, secondary control, tertiary control, power quality analysis, market & trading
load shedding (if generated power < demand), power quality mitigation, resynchronisation after islanding
Control algorithms in microgrids can be separated according to two axes: whether or not communication is involved (local vs. distributed) and whether or not real-time requirements need to be fulfilled. Table 1 provides examples of representative microgrid control applications algorithms. The distributed algorithms can be implemented as centralised (hierarchical) or decentralised (autonomous) systems.
However, the info’structure for microgrid control is not there yet. Generally speaking, communication and control systems that underpin electric power systems did not change significantly over the last 40 years; in spite of SCADA systems that became much more performant and computation power that has increased significantly, control remains largely centralised and several control loops contain human interference communicating via telephone, fax and email [10, 11]. Even more, humans can only process a limited amount of information, as a result of which more than 95 % of the captured data is dissolved in the aggregation process.
This paper reports on ongoing work to provide a dependable info’structure in this context, based on autonomous decentralised microgrid control where IEDs interact over a peer-to-peer overlay network, called Agora. Section 2 explains this approach to autonomous electricity grids, while section 3 elaborates on the resilience of the underlying overlay networks. Finally, section 4 outlines how we evaluate the approach.
2 Autonomous electricity grids
Electrical energy control applications can be considered as unbounded systems for which it is not possible to establish a global view at run-time. This is especially the case in a dispersed generation context where not all energy producers (wind, photovoltaics) or loads are available all the time; neither is it known beforehand which electrical loads, storages and generators will work together in a particular control application. In this context, it is relevant that the info’structure autonomously determines neighbours of an IED, and establishes communication links among them. An example in which small scale producers are involved in the control of the grid is the one of a microgrid, more specifically the Autonomous Electricity Grid. Such applications need to support not only static configurations, but also modifications during the application’s execution. Due to switching of generators and loads in a dispersed generation application, the components that need to communicate will for example vary in time, and hence, the logical communication topology has to follow accordingly.
To this extent, resource discovery is important, as to be able to quickly find the appropriate systems for the application at hand among the multitude of available systems. Several architectural configurations are possible for this resource discovery, such as a centralized or hierarchical indexing system, or a decentralized system in the form of an overlay network [5, 12]. In a microgrid context an overlay network or peer-to-peer network approach can be preferential, due to its inherent fault tolerance (no single point of failure), scalability, and automated management.
2.1 Semantic overlay network
The overlay network Agora, developed at K.U.Leuven, allows applications to query specific resources based on attributes defining the resources; hence a semantic overlay network [13]. The topology of these semantic networks is based on XML descriptions of resources, where neighbours of a single node are chosen based on a distance metric between its own XML description and the other node’s XML description. The smaller this distance, the larger the probability of the node becoming a direct neighbour. This topology allows to route attribute-queries based on these XML distances. As such, the logical topology of the semantic overlay network clusters IEDs with similar functionality (electricity meters, manageable loads, storage elements and generators, etc.). This is called group locality. Although this topology provides no deterministic query results, the overall efficiency is higher than unstructured overlay networks. Its query efficiency is lower than for deterministic overlay networks; but its added value is the broader range of supported applications, thanks to the functionality based organization and the resulting support of attribute-based semantic routing.
Such peer-to-peer network needs to periodically check for modifications: entities or links may appear, disappear or re-appear due to functional behaviour (no wind), due to electrical faults (short-circuits), or due to physical faults in the info’structure (controller or network breakdown). Indeed as parameters and functionality of entities change dynamically, so does the XML description describing these entities; hence, the overlay network needs to be adapted accordingly over time, to both retain its logical topology and to recover from errors. This ensures the time locality.
These semantic overlay networks fill the gap among existing decentralized resource discovery algorithms typically used in peer to peer systems [14], that is, the lack to search resources based on (a certain range of) values of multiple attributes:
Simulations show that semantic overlay networks, such as Agora, have a small-world property, meaning the average number of hops to reach any node from any other node is small (e.g. 4 to 5 hops) [13]. Small-world overlay networks find their likes in many networks in biology, technology and society, e.g., cellular networks, the worldwide web and citation networks [15].
It also incorporates several error detection and recovery techniques to deal with the partitioning (splitting) of the overlay network.
2.2 Gossiping for overlay communication
The main functionality the Agora overlay network offers is automatic resource discovery and the related semantic routing service with attribute-based addressing. As such, a structure is created for data and information aggregation, and for distributed cooperation and control among the IEDs. Besides message routing and resource discovery, this peer-to-peer network supports distributed control primitives, such as gossiping. Gossiping is a scalable distributed primitive for data dissemination and aggregation, based on the periodic exchange of status data by all devices with a randomly selected neighbour in the peer-to-peer network [16]. A low characteristic path length of the overlay network is required for efficient gossiping; this is obtained thanks to the small-world property.
Within Agora, every IED exchanges information at fixed time intervals with one of its neighbours (chosen randomly). If that neighbour exchanges this new information with one of its neighbours (and so forth), the news spreads in the network.
Using this basic communication paradigm, some basic functions can be implemented in overlay networks. One of these basic functions using gossiping based communication is distributed averaging: every node has a certain value (any real number) and using only gossiping, an overlay network wide average can be calculated. Such distributed averaging algorithm can be used in microgrid applications for secondary control (to maintain voltage and frequency within normal range). During gossiping the following steps happens: IED C1 IED C2 send current average Average1→C2
send current average Average2 → C1
receive average Average2 receive average Average1 calculate new average Ave.1→(Ave.1+Ave.2)/2
calculate new average Ave.2→(Ave.2+Ave.1)/2
Eventually, all IEDs will have the same value, equal to the average of all values.
2.3 Web services for power applications
Currently, we are extending this earlier work on unbounded overlay topologies in order to create a resilient application-related web service architecture in which microgrid electricity services function resiliently [17, 18], e.g. for trading load shedding contract. This requires decentralized resource discovery, dealing with an environment which is much larger (millions of loads and generators) and more dynamic (many loads are added and removed every minute, grid configuration are changed, etc.) than typical web services. It should be supported by flexible software modules at middleware level, one for each IED connected to the network, which periodically scan their environment and put up/break or modify connections. This provides a network abstraction to the applications allowing them to communicate based on their functionality (i.e. logical architecture), rather than on physical network topology.
3 Overlay resilience
3.1 Resilience against accidental faults
Small-world overlay networks are known to be quite resilient to errors [15, 19]. They can tolerate large numbers of node failures (10+%) without significant influence on the overlay’s regularity and small diameter or before breaking down into several partitions. Additionally, small-world systems have proven to be capable of automatic and swift adaptation to errors. This is due to the fact that no single node is crucial for the overlay network construction and maintenance; hence, no single points of failure exists. Secondly, overlay networks are built to deal with dynamic environments: new and/or leaving nodes, changing functionality or resource availability, etc. In fact, also errors represent a change to which the network must adapt. Since overlay networks incorporate the former, usually by means of self-organisation, they are well capable of the latter.
Within Agora, the self-organisation which leads to a high dependable system, can be tracked down to two algorithms; one providing error detection, the other providing error handling, which together results in graceful degradation in the advent of errors [13]. • An announcement mechanism ensures that each
node attempts to contact all its neighbours periodically, as to update its internal data structures to runtime description changes. Since every communication serves a secondary function as error detection mechanism, this puts an upper-bound on the error detection latency.
• Every node periodically also reconsiders its links in the overlay and reconverges as to adapt to topological changes elsewhere in the overlay. This same mechanism allows recovery from failed or unreachable nodes, posterior to their detection. The result is that Agora overlay networks establish smaller, yet internally optimised networks in the advent of errors. This graceful degradation may result in the
permanent splitting of the overlay into separate partitions. I.e., all links from one partition to the others are lost, which renders partitioning irreversible, even after repair of the errors that caused it. This is a problem all overlay networks suffer and, without extra measures, can only be solved by means of manually inserted cross-partition links. However, this process can also be made automatic, by ensuring that pointers from one partition to the other endure. As such, every Agora node maintains a small FIFO buffer of fixed size which contains the n addresses of the n last nodes that were detected as having failed. The result is that cross-partition pointers emerge, but also that pointers to failed nodes endure. In order to detect the recovery of previously failed nodes, each Agora node periodically attempts to contact the members of this list again. If this succeeds, a connection attempt is made to that recovered node, which consists of the transmission of a link request and a link announcement to the recovered node. Together, this constitutes the network merge detection algorithm. How often this algorithm must be invoked is a trade-off between reaction speed and network load. For Agora, the design choice was made to perform a network merge detection at the beginning of each periodic convergence cycle, as this is the point where the node already attempts to adapt to any changed network compositions with network merge detection being merely a different aspect of this. This periodic cycle is identical for all Agora nodes. Simulations with several hundreds of nodes confirmed the validity of this partition recovery [13].
3.2 Malicious fault resilience
Although overlay networks are inherently quite tolerant to accidental faults (redundancy in overlay links and automatic reconfiguration), malicious attacks can harm these networks significantly. Therefore, we have to look into the different known attack models on overlay networks (some are generic attacks on overlay networks, others specific for structured or unstructured networks), and see how we can make our approach more resilient against these kind of attacks. Many attack
models are based on the fact that there is an unbounded, relatively anonymous number of nodes in an overlay network, which complicates access control, and all these nodes have only a limited, local view on the system, which makes detection of large scale, coordinated attacks quite hard. • Denial of Service (DoS) attacks are a generic kind of
attack in open networks with shared bandwidth such as the IP-based Internet. These attacks can be aimed at knocking out the p2p application as well as any other application sharing the same bandwidth.
• Traffic amplification attacks are a way to perform a DoS attack, abusing p2p network protocols, namely the flooding protocol used for resource discovery by unstructured overlay networks. The attack is performed by sending a lot of queries to different nodes in the overlay network, which forward these queries to all their neighbours, which of course tremendously amplifies the original query-messages.
• Attacks abusing stabilization protocols are a different way to perform a DoS attack abusing p2p protocols, by directing at the stabilization protocols, which serve at optimizing a structured overlay topology when nodes enter or leave the network. These protocols consume a substantial amount of bandwidth. The attack is therefore performed by quickly entering and leaving the overlay network with several nodes, and doing so, constantly invoking the stabilization protocol.
• Bootstrapping attacks are based on the fact that any node in a p2p network has to enter the network using a random node already part of the network. In a bootstrapping attack, new nodes are tricked into using some malicious node as a bootstrapping node. The new node thinks it enters the p2p network, but in fact becomes part of a malicious counterpart without realizing it.
• Query falsification does not forward queries or returns false query results.
• Overlay topology attack abuse stabilization protocols in order to disturb the logical p2p topology. This may lead to more powerful (malicious) nodes, being able to falsify many queries or even to partition (split) the p2p network. A combination of intrusion prevention and intrusion
tolerance mechanisms will have to cope with these problems [18, 20].
4 Evaluation
In order to evaluate performance and dependability characteristics, a test bed has been developed which integrates the electric power system and the information infrastructure. It consists of several power electronic converters, which are electrically interconnected via a microgrid and whose IEDs are logically interconnected via off-the-shelf communication protocols (TCP/IP) [21]. Each converter can be used to emulate generators or loads in a dispersed electricity generation environment. This platform allows different control algoritms (see Table 1) to be modelled in a high level programming tool such as Matlab, after which it can be prototyped on a 4-quadrant power electronic converter, whereby the control algorithms are downloaded on high performance hardware (DSP + FPGA) which manages the power electronics. As these converters are connected to PCs, they can be interconnected via TCP/IP modules in order to extend the control scope from local towards hierarchical and decentralized control algorithms.
As a case study, distributed control in a microgrid has been evaluated. Besides the electrical connection between all generators, load and storage unit in a grid segment, these elements are also connected via the above-mentioned info’structure, based on the self-organizing semantic peer-to-peer network Agora [5]. At start-up, all entities broadcast some identification information (type, static and dynamic information) which results in the setup of a peer-to-peer network. On top of this communication overlay network control applications are run. Primary control is realized by means of an enhanced droop control [22], which requires no communication, thus guaranteeing a stable system, even when all communication fails. Secondary and tertiary control is performed by exploiting the peer-to-peer network. Secondary control consists of a gossiping-based distributed PI-controller, which keeps voltage and frequency into the correct range. The economic optimization or tertiary control is based on a variation of the averaging gossiping algorithm, using local generation cost-curves at each generator to re-dispatch the generated power, such that all operate at the same marginal cost.
This peer-to-peer-based solution includes two parallel control loops each containing an instantiation of the three levels of control: one for the voltage and active power balance maintenance and a second for the frequency and reactive power balance maintenance.
Experimental results confirm that the temporary or permanent unavailability of the communication links does not affect the control applications, as they are
handled at middleware level, only resulting in a negligible delay for the secondary and tertiary control algorithms that are not time-critical.
Malicious faults however, are more dangerous as they can lead to overvoltages which trigger the protection mechanisms in the platforms. Our future work will hence focus on integrating intrusion prevention and intrusion tolerance mechanisms in the middleware [18, 20].
Acknowledgements - This work was supported by the K.U.Leuven Research Council (GOA2007/09) and the European Union (IST 4-27513 CRUTIAL IST-4-026923 GRID).
References
[1] J. D. Kueck and B. J. Kirby, "The Distribution Grid of the Future," The Electricity Journal (Elsevier Science), pp. 78-87, Jun. 2003.
[2] G. Pepermans, J. Driesen, D. Haeseldonckx, R. Belmans, and W. D. Haeseleer, "Distributed generation: definition, benefits and issues," Energy Policy, vol. 33, pp. 787-798, 2005.
[3] M. C. Chandorkar, D. M. Divan, and R. Adapa., "Control of parallel connected converters in standalone ac supply systems," IEEE Trans. on Industry Applications, vol. 29, pp. 136-143, Jan.1993.
[4] M. N. Marwali, J.-W. Jung, and A. Keyhani, "Control of Distributed Generation Systems — Part II: Load Sharing Control," IEEE Trans. on Power Electronics, vol. 19, 2004.
[5] K. Vanthournout, K. D. Brabandere, E. Haesen, J. V. d. Keybus, G. Deconinck, and R. Belmans, "Agora: Distributed tertiary control of distributed resources," in 15th Power Systems Computation Conference (PSCC15), Liege, Belgium, 2005, p. 7.
[6] M. Adamiak and W. Premerlani, "Data communications in a deregulated environment," IEEE Computer Applications in Power, vol. 12, pp. 36-39, Jul. 1999.
[7] G. Dondossola, J. Szanto, M. Masera, and I. N. Fovino, "Evaluation of the effects of intentional threats to power substation control systems," in Proceeding of the International Workshop on Complex Network and Infrastructure Protection (CNIP-2006) Rome, Italy, Mar 2006, pp. 309-320.
[8] M. Amin, "Towards self-healing energy infrastructure systems," IEEE Computer Applications in Power, vol. 14, pp. 20-28, Jan. 2001.
[9] G. Deconinck, V. De Florio, and O. Botti, "Software-implemented fault-tolerance and separate recovery strategies enhance maintainability [substation automation]," Reliability, IEEE Transactions on, vol. 51, pp. 158-165, 2002.
[10] C. H. Hauser, D. E. Bakken, and A. Bose, "A Failure to Communicate," IEEE Power & Energy Magazine, vol. 3, pp. 47-55, Mar. 2005.
[11] A. Stefanini and A. Servida, "The future of ICT for power systems: emerging security challenges," in Joint DG INFSO, DG RTD and JRC workshop on R&D challenge, Brussels, Belgium, Feb. 2005.
[12] K. Vanthournout, G. Deconinck, and R. Belmans, "A middleware control layer for distributed generation systems (on CD Rom)," in IEEE Power Systems Conference and Exhibition (PSCE), New York City, USA, 2004, p. 5.
[13] K. Vanthournout, "A semantic overlay network based robust data-infrastructure, applied to the electric power grid." vol. PhD: K.U.Leuven, ESAT-ELECTA 2006.
[14] K. Vanthournout, G. Deconinck, and R. Belmans, "A Taxonomy for Resource Discovery," Personal and Ubiquitous Computing Journal (Springer), vol. 9, pp. 81-89, 2005.
[15] R. Albert and A.-L. Barabasi, "Statistical mechanics of complex networks," Reviews of Modern Physics, vol. 74, pp. 47-97, Jan 2002.
[16] M. Jelasity, A. Montresor, and O. Babaoglu, "Gossip-based aggregation in large dynamic networks," ACM Transactions on Computer Systems, vol. 23, pp. 219-252, Aug. 2005.
[17] T. Rigole, K. Vanthournout, and G. Deconinck, "Distributed control systems for electric power applications," in 2nd International Workshop on Networked Control Systems: Tolerant to Faults, Rende (CS), Italy, 2006, p. 7.
[18] G. Dondossola, G. Deconinck, F. D. Giandomenico, S. Donatelli, M. Kaaniche, and P. Verissimo, "Critical Utility InfrastructurAL Resilience," in Int. Workshop on Complex Network and Infrastructure Protection (CNIP-2006), Rome, Italy, 2006, p. 4.
[19] R. Albert, H. Jeong, and A. Barabasi, "Error and attack tolerance of complex networks," Nature, vol. 406, pp. 378-382, Jul 2000.
[20] P. Verıssimo, N. F. Neves, and M. Correia, "CRUTIAL: The Blueprint of a Reference Critical Information Infrastructure Architecture " in 1st International Workshop on Critical Information Infrastructures Security (CRITIS-2006) Samos, Greece, 2006, p. 6.
[21] J. V. d. Keybus, B. Bolsens, K. D. Brabandere, and J. Driesen, "Using a fully digital rapid prototype platform in grid-coupled power electronics applications," in 9th IEEE Conference on Computers and Power Electronics (COMPEL), Urbana-Champaign, USA, 2004, p. 10.
[22] K. D. Brabandere, B. Bolsens, J. V. d. Keybus, A. Woyte, J. Driesen, and R. Belmans, "A voltage and frequency droop control method for parallel inverters," in IEEE 35th Annual power electronics specialists conference (PESC), Aachen, Germany, 2004, pp. 2501-2507.