final report version ii on the total power system failure

1

Final Report Version II on the

Total Power System Failure on

17th August 2020

Submitted to the Ministry of Power on Wednesday the 16th December 2020

NOTE:

THE AUTHOR OF THIS VERSION II OF THE FINAL REPORT HAS BEEN A MEMBER OF THE COMMITTEE

APPOINTED BY THE MINISTRY OF POWER IN ITS LETTER DATED 18.08.2020 (REF:

PE/DEV/01/81/2015). HE DOES NOT AGREE WITH SOME OF THE TECHNICAL INTERPRETATIONS

GIVEN BY SOME MEMBERS IN THE COMMITTEE AND THEREFORE PRESENTED THIS VERSION II AS A

SEPARATE DOCUMENT.

2

DISCLAIMER

The Conclusions and Recommendations stipulated herein have been framed based on the information

ascertained and discussions held with relevant officials of the Ceylon Electricity Board (CEB) through

number of visits made to Kerawalapitiya GSS, Lak Vijaya Power Station (LVPS), Protection Branch and

the System Control Center (SCC) of CEB. As such the professional judgment of the individual in making

the Conclusions and the Recommendations would depend on the accuracy of the said information

ascertained and those transpired from the said information and discussions. Further, the author would

not be responsible for the extrapolation of the Conclusions and the Recommendations stipulated

herein by any third party including the other members of the committee appointed for the same

purpose.

3

Contents

Introduction ………………………………………………………………………………………………………………………….…….4

1. Kerawalapitiya Grid Substation ..………………………………………………………………………………………..…5

1.1 Relevant observations and findings during the visit to Kerawalapitiya Grid Substation (GSS)

on the 19.08.2020

1.2 Analysis of the observations made during the visit to Kerawalapitiya GSS

2 Protection Branch / CEB .…………………………………………………………………………………………………6

2.1 Relevant observations and findings during the meeting with the Control and Protection

Branch Engineers on the 19.08.2020

2.2 Analysis of the observations made during the meeting Control and Protection Branch

Engineers

3. Lak Vijaya Power Station (LVPS) ……………………………………………………………………………………10

3.1 Relevant findings during the visit to the LVPS on the 20.08.2020

3.2 Analysis of the observations made during the visit to LVPS 3.2.1 Timing Data Analysis

3.2.2 Fault Current Analysis

3.2.3 System Frequency Analysis

3.2.4 Distributed Control System Data Analysis

3.2.5 LVPS Generator Current Analysis

3.2.6 Fast Cut Back (FCB) operation at the LVPS

3.2.7 Time sequence analysis of Over Speed Protection Control (OPC) at the LVPS

3.3 Observations on the LVPS behavior in this incident

3.4 Root Cause of the incident related to LVPS

4. System Control Centre / CEB ………………………………………………………………………………………..30

4.1 Relevant findings of the committee during the visit to the System Control Office (visited on

the 22nd August 2020)

4.2 Analysis of the observations made during the visit to System Control

4.3 Observations on the CCTV records at the Systems Control Center

5. Conclusions ..……………………………………………………………………………………………………………….34

6. Recommendations ..…………………………………………………………………………………………………….35

4

Introduction

This report has been prepared by a single author in the committee appointed by the Secretary to the

Ministry of Power in order to investigate the nationwide electricity supply interruption took place on

the 17th August 2020.

A nationwide electricity supply interruption had taken place around 12.30 pm on the 17th August 2020,

involving all Transmission lines, Power Stations and Grid Substations in the Power System in Sri Lanka,

thereby disconnecting all the electricity consumers of the country from the electricity power supply.

The said interruption is reported to have been initiated at Kerawalapitiya Grid Sub Station triggering

a total power system failure in the country. Consequent restoration of the transmission network, in

spite of all efforts initiated and managed by the System Control Center of the CEB with the nodal

station of the utilities had taken 6 hours and 16 minutes.

5

1. Kerawalapitiya Grid Substation

1.1 Relevant observations and findings during the visit to Kerawalapitiya Grid Substation (GSS) on the

19.08.2020

Routine maintenance work on the 220 kV isolators of the Bus Coupler Bay had been carried out on the

day of the incident by the Electrical Superintendent In Charge at Kerawalapitiya GSS, who apparently

has been attending routine maintenance work at the Kerawalapitiya GSS for the past 5 years. The

power in the Bus Bar 01 had been turned OFF for the maintenance, while the power of the Bus Bar 02

was ON. The Earth Switch 01 at Bus Bar 01 side had been OFF while the Earth Switch 02 at Bus Bar 02

side had been ON as shown in Fig. 1.2(a) at the time of incident. Under normal operations the Earth

Switch and the relevant isolator are interlocked, so that the isolator cannot be turned ON while the

Earth Switch is turned ON. However, during maintenance, this interlock had been bypassed, so that

isolator can be turned ON even with the Earth Switch is turned ON. At the end of the maintenance

work of the 220 kV Bus Coupler Bay, while the interlock is bypassed, the Isolator on the Bus Bar 02

side had been turned ON as shown in Fig. 1.2(b), creating a 3 Phase to Ground fault. The fault had

been isolated within 154 ms by tripping of all connected circuit breakers to the Bus Bar 2 by the

operation of Busbar differential protection.

It was revealed that the Electrical Superintendent In Charge at Kerawalapitiya GSS had been carrying

out the above maintenance work by himself with his team around. It was further revealed that this

maintenance work had been carried out without being supervised by any of his superiors, which is the

usual practice.

Bus bar 01: OFF

Bus bar 02: ON

Earth Switch 1

Earth Switch 2

Isolator 1

Isolator 2

Bus coupler

Fig. 1.2a

Bus bar 02: ON

Earth Switch 1

Earth Switch 2

Isolator 1

Isolator 2

Bus coupler

Bus bar 01: OFF

High fault current flow

Fig. 1.2b

1.2 Analysis of the observations made during the visit to Kerawalapitiya GSS

The total isolation time of 154 ms is too long on the busbar protection operation. As per the report of

Control & Protection branch also, the time to issue the trip command is 87 ms. Therefore, the fault

detection time of the busbar protection relay has to be checked. Turning ON the Isolator 2 could be a

human error (mistake), which should have been the Isolator 1 instead. However, if not for bypassing

the interlocking, a catastrophe by such a human error could have been avoided.

It is also noted that there is no robust maintenance protocol in place. The maintenance practices of

high risk areas are to be identified and best practiced protocols in maintenance to be incorporated in

particular in the said areas rather than having a general maintenance philosophy.

6

2. Protection Branch / CEB

2.1 Relevant observations and findings during the meeting with the Control and Protection Branch

Engineers on the 19.08.2020

During the meeting, the Protection Branch Engineers showed various data associated with the

incident using the data acquisition system in the CEB. Out of them, the bus voltage data shows

that the 3 Phase to Ground fault had been so severe that for 7.5 cycles, the 220 kV bus voltage

had been reduced to

1. 0V according to DDR record of Sub L bay of 220kV Kerawalapitiya GIS at 12:30:27 Hrs

2. 110 kV according to DDR record of Generator Transformer 2 bay of 220kV Lakvijaya GIS at

12:30:27 Hrs respectively. The corresponding voltages are reproduced from DDR in Fig. 2.1a

and 2.1b respectively.

Fig. 2.1a

220 kV reduced to 0 kV

7

Fig. 2.1b

Accordingly, in addition to the busbar voltage becoming zero at the fault location, reducing the

voltages to 50% at far away bus bars such Lakvijaya Power Station Norechcholei shows the severity

of the fault.

Further, the Disturbance Recorders (DDR) in the Control and Protection Branch / CEB shows

1. Busbar protection at Kerawalapitiya 220kV GIS operated and tripped all connected

Transmission Lines and Transformers by at 12:30:27.172 Hrs, due to a busbar fault.

2. 220kV Circuit breaker of Generator Transformer 3 at Lakvijaya Power Station tripped at

12:30:27.359 Hrs

3. 220kV Circuit breaker of Generator Transformer 2 at Lakvijaya Power Station tripped at

12.30.27.390 Hrs

4. 220kV Circuit breaker of Generator Transformer 1 at Lakvijaya Power station tripped at

12.30.27.429 Hrs

5. Phase B to Ground fault of approximately 8.5 kA occurred in 220kV Bus section 1/3 at

Lakvijaya Power Station 12:30:27.423 Hrs and all lines and Transformers connected to

Bus Section 1 and Bus section 3 tripped due to operation of busbar protection by

12:30:27.491 Hrs

6. As per the DDR of Kotmale PS, the system frequency dropped below 47 Hz within 1.9s

from the initial Busbar Protection fault at Kerawalapitiya Grid Substation.

2.2 Analysis of the observations made during the meeting Control and Protection Branch Engineers

Fig. 2.2a shows the transients of the bus bar voltages during and immediately after the fault isolation

in the nearby bus bars. Accordingly, Kerawalapitiya Bus bar 02 voltage drops from 220 kV to 0 kV

during the 3 phase to ground fault because the fault occurs through near zero resistance conductor

and does not recover because the busbar is isolated from the system by the operation of the busbar

220 kV reduced to ~110 kV

8

protection. The Kotugoda Circuit 01 (220 kV), which is the next node from Kerawalapitiya GSS, also

undergoes the same voltage drop. Though it recovers, it does not settle. Similarly, in the 132 kV

network also Kolonnawa GSS experience a drop from 132 kV to about 40 kV and many other places

experience similar drops. In far away places from the fault such as the Veyangoda GSS, the Generator

Transformer 2 bay of 220kV Lak Vijaya Power Station (Fig. 2.1 b), the 220 kV busbar voltage drops to

about 110 kV. The active power 𝑃 at a busbar is proportional to the voltage 𝑉𝐿𝐿 squared.

Fig. 2.2 a

The apparent power 𝑆 at a node can be expressed by

𝑆 = 𝑃 + 𝑗𝑄 =√3𝑉𝐿𝐿

2

𝑅 + 𝑗𝑋=

√3𝑉𝐿𝐿2

(𝑅2 + 𝑋2)2(𝑅 − 𝑗𝑋)

If the overall voltage drop is assumed to be 50% during the fault, the active power in the system

reduces to 25% of the pre fault value, which corresponds to a loss of load by 75%. The corresponding

dynamics can be modelled by

𝜏𝑔𝑒𝑛 − 𝜏𝑙𝑜𝑎𝑑 = 𝐽𝑑𝑓

𝑑𝑡+ 𝐵𝑓.

Where 𝜏𝑔𝑒𝑛represents generation, 𝜏𝑙𝑜𝑎𝑑 represents load, 𝐽 is the system inertia, 𝐵 system damping

and 𝑓 is system frequency, a drop in 𝜏𝑙𝑜𝑎𝑑 causes an increase in the difference in 𝜏𝑔𝑒𝑛 − 𝜏𝑙𝑜𝑎𝑑 which

gives rise to 𝑑𝑓

𝑑𝑡. Hence the rate of change of system frequency depends on the system inertia. If there

is adequate system inertia, the rate of change of system frequency can be kept within the accepted

limits.

0

50

100

150

200

250

12

:30

:26

.18

8

12

:30

:26

.28

8

12

:30

:26

.38

8

12

:30

:26

.48

8

12

:30

:26

.58

8

12

:30

:26

.68

8

12

:30

:26

.78

8

12

:30

:26

.88

8

12

:30

:26

.98

8

12

:30

:27

.08

8

12

:30

:27

.18

8

12

:30

:27

.28

8

12

:30

:27

.38

8

12

:30

:27

.48

8

12

:30

:27

.58

8

12

:30

:27

.68

8

12

:30

:27

.78

8

12

:30

:27

.88

8

12

:30

:27

.98

8

12

:30

:28

.08

8

12

:30

:28

.18

8

12

:30

:28

.28

8

12

:30

:28

.38

8

12

:30

:28

.48

8

12

:30

:28

.58

8

12

:30

:28

.68

8

12

:30

:28

.78

8

12

:30

:28

.88

8

12

:30

:28

.98

8

12

:30

:29

.08

8

12

:30

:29

.18

8

12

:30

:29

.28

8

12

:30

:29

.38

8

Bu

s b

ar v

olt

age

(kV

)

Time

Bus bar voltages in the Vicinity of Kerawalapitiya GSS during the fault and immediately after fault

Kearawalapitiya WCP 02 (220kV)

Kotugoda Circuit 01 (220kV)

Kolonnawa (132kV)

9

The above reduction in voltage reduces the power to 25% of the power flowing before the initiation

of the fault, meaning that the grid feels a loss of 75% of the load. The load at the time of incident had

been 1983 MW and loosing 75% of it gives the grid, a feeling similar to a single generator suddenly

loosing 75% of its load. In the single generator case, the generator frequency increases. Similarly, in

this case, the system frequency should increase. Once the fault is cleared, the frequency should stop

increasing and settle. However, the above scenario leads to a total system failure and therefore the

system frequency should start reducing. The system frequency waveform against time obtained by

the Control and Protection Branch / CEB at 12:30:27 hrs on 17.08.2020 at Kotmale Power Station,

reproduced in Fig. 2.2b confirms this hypothesis.

Fig.2.2b System frequency variation during the system failure

10

3. Lak Vijaya Power Station (LVPS)

3.1 Relevant findings during the visit to the LVPS on the 20.08.2020

The Power House, Switch Yard and the Control Centre of the LVPS were visited. During the visit,

the Switchyard layout, Voltage, Current, Power, Generator Frequency, System Frequency, Turbine

speed plots and the operation of the digital logic circuits of the relevant protection relays against

time, pre and post of the system power failure were inspected. Fig. 3.1a shows the Busbar

arrangement of the LVPS 220 kV GIS during the fault.

Fig. 3.1 a Busbar arrangement of the LVPS 220 kV GIS during the fault

Following elements have been reported in the report.

(1). Busbar protection at Kerawalapitiya 220kV GIS operated and tripped all connected Transmission

Lines and Transformers by at 12:30:27.172 Hrs, due to the three phase to earth fault in the Bus

coupler.

(2). 220kV Circuit breaker of Generator Transformer 3 at Lakvijaya Power station tripped at

12:30:27.359 Hrs


12.30.27.390 Hrs


12.30.27.429 Hrs

(5). Phase B to Ground fault of approximately 8.5 kA occurred in 220kV Bus section 1/3 at 12:30:27.423

Hrs and all lines and Transformers connected to Bus Section 1 and Bus section 3 tripped due to

operation of busbar protection by 12:30:27.491 Hrs.

(6). Lakvijaya-New Chilaw Line 01 and Lakvijaya-New Anuradhapura Line 02 have been tripped by

busbar protection at 12:30:27.478 hrs and 12:30:27.480 hrs respectively while Lakvijaya-New Chilaw

Line 02 and Lakvijaya-New Anuradhapura Line 01 were not tripped.

11

(6). As per the DDR of Kotmale PS, the system frequency dropped below 47 Hz within 1.9s from the

earth fault at Kerawalapitiya Grid. Frequency variation at the LVPS during the system failure,

reproduced from the report of CEB Protection and Control Branch, is shown in the Fig. 3.1b.

(7). Generator Transformers 3, 2 and 1 of Lakvijaya Power Station have tripped in 180ms, 210ms and

246ms respectively after the Kerawalapitiya three phase to earth fault has been cleared.

(8). Another Phase B - N fault has occurred in Bus Section 1/3 resulting in the operation of Busbar

protection in both Bus 1 and Bus 3. The Busbar section fault was cleared within 61ms. However, Bus

2 remained stable until the system was collapsed.

According to the above DDR records, it can be seen that Generator transformer CB of Unit 3 has been

tripped first and Unit 2 CB and Unit 1 CB have been tripped second and third following the earth fault

at Kerawalapitiya GSS.

Fig. 3.1b Frequency variation at the Lakvijaya Power Plant during the system failure.

12

3.2 Analysis of the observations made during the visit to LVPS

3.2.1 Timing Data Analysis

Table 3.2a shows the time records of relevant events at LVPS

TABLE 3.2a

Unit df/dt Generator Transformer Circuit Breaker Opening Time

Unit Shutdown Time

01 12:30:27:226.4 12:30:27:425.4 12:42

02 12:30:27:213.71 12:30:27:395.38 14:51

03 12:30:26:508.25 12:30:26:682 13:33

Since the system frequency is common to the entire grid, all three units should detect the system

frequency rise at the same time. However, as per the records in TABLE 3.2a, they are at different

instants in time, and hence the DFR clocks are out of synchronism. The difference between df/dt

record of Unit 01-Unit 03, which is (1000-508.25) +226.4 = 718.15 ms, which is approximately the time

synchronization error at the time of this recording.

Not having GPS synchronized clocks at all locations in the system has made the analysis some what

complicated.

3.2.2 Fault Current Analysis

According to the officials of the LVPP and the Transmission Protection branch, Kerawalapitiya three

phase fault has been picked up by the three generator units at LVPP as an external fault, when the

rate of change of frequency (df/dt) of the system increased beyond the currently set thresholds. This

df/dt threshold limit has been set to 2 Hz/s for all the three units at the LVPP. Consequently, the

generator df/dt protection scheme (ROCOF relay) has operated and tripped the generator transformer

circuit breakers. A phase to earth fault is also observed in the 1-3 bus section in the LVPS, after the

Kerawalapitiya three phase fault cleared. It is noted that, in the ROCOF relay, only df1/dt signal is

used to trip the Generator transformer circuit breakers while df2/dt, the other signal is currently not

in use. Apart from the indication about an advice provided by a Consultant to consider only df1/dt,

there is no proper evidence nor details of df/dt signal established at the commissioning stage were

made available. As per the LVPP officials, generator transformer Circuit Breaker (CB) tripping sequence

observed of the Units was Unit 03, Unit 02 and Unit 01 and the three units had been put on the Fast

Cut Back (FCB) operation and generator units had been put into the house load operation (Unit 01 -

12 minutes, Unit 02 – 2 hours and 20 minutes, Unit 03 – 1 hour), before the units shutdown. This

sequence of units going into FCB can be observed in the generator current waveforms in Fig. 3.2a,

3.2b and 3.2c whose data obtained from the BEN6000 recorder at LVPS which sampled at 200 µs.

13

Fig. 3.2a Current waveforms of Phase R of LVPS units during and after the Three Phase to Ground fault

at Kerawalapitiya GSS. Accordingly, Unit 03 current (Ash colour waveform) reduces first, Unit 02

current (Orange colour waveform) reduces next and the Unit 01 current (Blue colour waveform)

reduces last.

Fig. 3.2b Current waveforms of Phase Y of LVPS units during and after the Three Phase to Ground fault



reduces last.

-60000

-50000

-40000

-30000

-20000

-10000

0

10000

20000

300001

96

19

1

28

6

38

1

47

6

57

1

66

6

76

1

85

6

95

1

10

46

11

41

12

36

13

31

14

26

15

21

16

16

17

11

18

06

19

01

19

96

20

91

21

86

22

81

23

76

24

71

25

66

26

61

27

56

28

51

29

46

30

41

31

36

32

31

33

26

34

21

Zoomed Generator Phase R Currents

Serial number : 20 GEN 1_IR A Serial number : 20 GEN 2_IR A Serial number : 20 GEN 3_IR A

-25000

-20000

-15000

-10000

-5000

0

5000

10000

15000

20000

25000

1

96

19

1

28

6

38

1

47

6

57

1

66

6

76

1

85

6

95

1

10

46

11

41

12

36

13

31

14

26

15

21

16

16

17

11

18

06

19

01

19

96

20

91

21

86

22

81

23

76

24

71

25

66

26

61

27

56

28

51

29

46

30

41

31

36

32

31

33

26

34

21

Zoomed Generator Phase Y Currents

20 GEN 1_IY A 20 GEN 2_IY A 20 GEN 3_IY A

14

Fig. 3.2c Current waveforms of Phase B of LVPS units during and after the Three Phase to Ground fault



reduces last.

Hence with the synchronous sampled data, sampled at 200 µs by BEN6000 fault recorder at LVPS, it

is confirmed that the sequence of LVPS Generator Units going into FCB mode is Unit 03, Unit 02 and

Unit 01, irrespective of what can be derived using logic signals in DCS recorded at low time

resolutions in the range of 100 – 200 ms, which is three orders of magnitude lower in resolution.

The following parameters in TABLE 3.2b and 3.2c are used to calculate the fault currents of the LVPS

generators during the Three Phase to Ground fault at Kerawalapitiya GSS.

TABLE 3.2b

Line Section

No. of

Circuits

Cable type Distance

(km)

R (/km) X (/km)

Norochcholi - New Chillaw 2 AAAC 630 73.3 0.02378 0.2818

New Chillaw - Veyangoda 2 AAAC 400 44.2 0.03662 0.2961

Veyangoda – Kotugoda 2 Zebra 17.0 0.03807 0.306

Kotugoda – Kerawalapitiya 2 Zebra 18.0 0.03807 0.306

By using 100 MVA and 220 kV base, the impedance base is Z base = 2202/100 = 484 . The per unit

quantities of data in Table 3.2b are given in Table 3.2c.

-30000

-20000

-10000

0

10000

20000

30000

40000

50000

600001

96

19

12

86

38

14

76

57

16

66

76

18

56

95

11

04

61

14

11

23

61

33

11

42

61

52

11

61

61

71

11

80

61

90

11

99

62

09

12

18

62

28

12

37

62

47

12

56

62

66

12

75

62

85

1

29

46

30

41

31

36

32

31

33

26

34

21

Zoomed Generator Phase B Currents

Sampling Rate : 20 GEN 1_IB A Sampling Rate : 20 GEN 2_IB A Sampling Rate : 20 GEN 3_IB A

15

TABLE 3.2c

Line Section R () X () R (pu) X (pu)

Norochcholi - New Chillaw 1.74 20.66 0.0036 0.0427

New Chillaw - Veyangoda 1.62 13.09 0.0033 0.0271

Veyangoda – Kotugoda 0.64 5.20 0.0013 0.0107

Kotugoda – Kerawalapitiya 0.69 5.51 0.0014 0.0114

Generator transformer reactance = 0.14 pu on 360 MVA from the datasheet and it is 0.0389 on the

100 MVA base. As can be seen in Fig. 3.2d the RMS value of the fault current on 20 kV side is 10.094

kA.

Fig. 3.2d Circuit from LVPS to Kerawalapitiya GSS

Table 3.2d shows the RMS fault currents calculated from the measured currents in BEN6000 recorder.

TABLE 3.2d RMS fault currents calculated from measurements

Generator Unit 01

Generator Unit 02

Generator Unit 03

Phase R 10394 A 10499 A 10574 A

Phase Y 10600 A 10635 A 10722 A

Phase B 9415 A 9514 A 9613 A

Hence the calculated fault currents are in good agreement with the measurements despite possible

line parameter mismatches. It confirms that no incident has taken place at LVPS during the three

phase to ground fault at Kerawalapitiya GSS.

3.2.3 System Frequency Analysis

In case of an external fault, the Rate Of Change Of Frequency (ROCOF) is picked up by the Generator

df/dt protection incorporated in the ROCOF relay (81R in SIEMENS 7SJ6225). As the upper threshold

limit of df/dt has been set to 2Hz/s for all three units identically, in the case of an external fault, if the

df/dt limit exceeds the said threshold, the generator transformer CBs of all three units will trip

initiating the Fast Cut Back (FCB) operation and subsequently the generator units will be put into the

house load operation, usually for about 30 mins or so, expecting the grid power to put the units back

into operation.

16

According to the manual of ROCOF relay 81R of SIEMENS 7SJ6225, the measuring principle is (quoted

from the manual) “From the positive-sequence voltage, the frequency is determined once per cycle

over a measuring window of 3 cycles, and a mean of two consecutive frequency measurements is

formed. The frequency difference is then determined over a settable time interval (default setting

5 cycles). The ratio between frequency difference and time difference corresponds to the frequency

change; it can be positive or negative. The measurement is performed continuously (per cycle).

Monitoring functions such as undervoltage monitoring, check for phase angle jumps etc. help to

avoid malfunctioning.”

Since the only unknown here is the default setting which could be at least 2 cycles and assuming that

it does not go beyond 5 cycles due to long response times, it is varied from 2 cycles to 5 cycles and

df/dt has been calculated on the frequency data on the Kolonnawa 132 kV busbar sampled at 20 Hz

(sampling period of 50 ms) and is plotted in Fig. 3.2e. It shows the rate of change of the grid frequency

(df/dt) during the Three Phase to Ground fault at Kerawalapitiya GSS.

Fig. 3.2e Rate of change of frequency for different cycle times.

As can be observed, despite it is 20 Hz sampled dataset, for the 2 cycles average, 3 cycles average,

4 cycles average and 5 cycles average, the df/dt reaches above 2 Hz/s. Since the sampling itself acts

as a filter, a dataset sampled at a higher rate than 20 Hz (theoretically it can go up to 100 Hz, if zero

crossing method is used to find the frequency, but at the expense of accuracy) would result in even

higher figures for df/dt shown above. Hence it is concluded that the rate of system frequency has

reached 2 Hz/s which is sufficient to trigger the ROCOF threshold who is set to operate at 2 Hz/s.

-4

-3

-2

-1

0

1

2

3

4

1 3 5 7 9 111315171921232527293133353739414345474951535557596163656769717375777981838587

df/dt

2 cycle df/dt 3 cycle df/dt 4 cycle df/dt 5 cycle df/dt

17

3.2.4 Distributed Control System Data Analysis

In the High-Speed Recorders (HSR) of the respective Distributed Control Systems (DCS) in the Control

Centre of the LVPS, the Turbine Speed, the Generator Power, the Generator frequency and the rate

of change of frequency 𝑑𝑓

𝑑𝑡 recordings were studied. Since the sampling times used in the Unit 01, 02

and 03 DCS are as large as 105 ms, 200 ms and 105 ms respectively, the HSR records have only be used

to confirm if a certain event has taken place or not. Otherwise, they cannot be used to derive the time

sequence of events.

LVPS Unit 03 (Sampling period = 105 ms): According to Fig. 3.2f obtained from the Unit 03 DCS, the

FCB comes into operation following the activation of the 𝑑𝑓

𝑑𝑡, resulting in a drastic reduction in the

generated power, which causes an overshoot in the turbine speed.

Fig. 3.2f The activation of the 𝑑𝑓

𝑑𝑡 of Unit 03

LVPS Unit 02 (Sampling period = 200 ms): According to Fig. 3.7 obtained from the Unit 02 DCS, FCB

comes into operation, following the activation of the 𝑑𝑓

𝑑𝑡. This is observed in the drastic reduction in

the generated power and the overshoot in the turbine speed.

Fig. 3.2g The activation of the 𝑑𝑓

𝑑𝑡 of Unit 02

2800

2850

2900

2950

3000

3050

3100

3150

3200

0

50

100

150

200

250

300

350

1

22

43

64

85

10

6

12

7

14

8

16

9

19

0

21

1

23

2

25

3

27

4

29

5

31

6

33

7

35

8

37

9

40

0

42

1

44

2

46

3

48

4

50

5

52

6

54

7

56

8

58

9

Sample

Generator Unit 03 HSR Data

Load (MW) df/dt FCB Breaker Open Speed (rpm)

2800

2850

2900

2950

3000

3050

3100

3150

3200

0

50

100

150

200

250

300

350

1

21

41

61

81

10

1

12

1

14

1

16

1

18

1

20

1

22

1

24

1

26

1

28

1

30

1

32

1

34

1

36

1

38

1

40

1

42

1

44

1

46

1

48

1

50

1

52

1

54

1

56

1

58

1


Load (MW) FCB df/dt Breaker Open Speed (rpm)

18

LVPS Unit 01 (Sampling period = 105 ms): According to Fig. 3.2h obtained from Unit 01 DCS, the 𝑑𝑓

𝑑𝑡 of

Unit 01 is activated which is confirmed by the overshoot in the Generator Frequency and the

overshoot in the Turbine Speed. This activation has reduced the power and tripped the 220kV Circuit

breaker of Generator Transformer 01 of the Unit 01.

Fig. 3.2h The activation of the 𝑑𝑓

𝑑𝑡 of Unit 01

3.2.5 LVPS Generator Current Analysis

The data in the BEN6000 at LVPS were synchronous sampled at a period of 200 µs and the first 2700

data points of the data records provided by the LVPS officials were used to extract the LVPS Generator

Unit current information shown in Figs. 3.2i, 3.2j and 3.2k. The GPS synchronized BEN6000 fault

recorder records data in a window from 100 ms before the fault to 2500 ms after triggering from the

fault. It is clear in them that there has not been any fault incident prior to the Three Phase to Ground

fault at Kerawalapitiya GSS. The fault at Kerawalapitiya GSS clears in nearly 7.5 cycles, i.e.,

approximately 150 ms, which is in agreement with the recorded fault clearance time of 154 ms

recorded and reported earlier. The screen shot in Fig. 3.2m confirms this recording window

information. Accordingly, for the Three Phase to Ground Fault at Kerawalapitiya GSS took place at

12:30:27.024 GPS, the BEN6000 at LVPS can only show data back to 12:30:26.924 GPS in relation to

the incident at Kerawalapitiya GSS.

0

500

1000

1500

2000

2500

3000

3500

0

50

100

150

200

250

300

350

Sam

ple 18

36

54

72

90

10

8

12

6

14

4

16

2

18

0

19

8

21

6

23

4

25

2

27

0

28

8

30

6

32

4

34

2

36

0

37

8

39

6

41

4

43

2

45

0

46

8

48

6

50

4

52

2

54

0

55

8

57

6

59

4


Load (MW) Breaker Open df/dt FCB Frequency (Hz) Speed (rpm)

19

Fig. 3.2i BEN6000 record of currents of the three phases of Generator Unit 01 at LVPS: Sequentially

from left to right it shows (i) before the three phase to ground fault at Kerawalapitiya GSS, (ii) during

the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) increase due to taking up

the load when Unit 03 leaving the grid and going to FCB, (v) increase due to taking up the load when

Unit 02 leaving the grid and going to FCB, (vi) Fault current in R and B phases, (vii) reduced current

due to entering FCB.

Fig. 3.2j BEN6000 record of currents of the three phases of Generator Unit 02 at LVPS: Sequentially


the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) increase due to taking up

the load when Unit 03 leaving the grid and going to FCB, (v) reduced current due to entering FCB.

-50000

-40000

-30000

-20000

-10000

0

10000

20000

30000

40000

500001

74

14

7

22

02

93

36

6

43

9

51

2

58

5

65

8

73

1

80

4

87

7

95

0

10

23

10

96

11

69

12

42

13

15

13

88

14

61

15

34

16

07

16

80

17

53

18

26

18

99

19

72

20

45

21

18

21

91

22

64

23

37

24

10

24

83

25

56

26

29

Generator Unit 01 Currents

20 GEN 1_IR A 20 GEN 1_IY A 20 GEN 1_IB A 20 GEN 1_IN A

(ii)(iii) (iv)

(v)

(vi)

(vii)

-25000

-20000

-15000

-10000

-5000

0

5000

10000

15000

20000

25000

1

74

14

7

22

02

93

36

6

43

9

51

2

58

5

65

8

73

1

80

4

87

7

95

0

10

23

10

96

11

69

12

42

13

15

13

88

14

61

15

34

16

07

16

80

17

53

18

26

18

99

19

72

20

45

21

18

21

91

22

64

23

37

24

10

24

83

25

56

26

29



(i)

(ii)

(iii)(iv)

(v)

(i)

20

Fig. 3.2k BEN6000 record of currents of the three phases of Generator Unit 03 at LVPS: Sequentially


the fault at Kerawalapitiya GSS, (iii) after fault at Kerawalapitiya GSS, (iv) reduced current due to the

Unit 03 entering FCB.

Towards the right end of Fig. 3.2i, it shows a huge current in the order of 50 kA in the R and B phases

of Generator Unit 01. Since the 20 kV winding side of the Generator Transformer is connected to the

Generator Stator windings is delta connected, it appears as a phase to phase fault current. However,

when the Generator Transformer 220 kV winding side, whose star point is solidly grounded, is

observed as shown in Fig. 3.2l, it is a Phase B to Ground (N) fault after the Generator Unit 01 going to

FCB.

Fig. 3.2l

-25000

-20000

-15000

-10000

-5000

0

5000

10000

15000

20000

250001

74

14

7

22

02

93

36

6

43

9

51

2

58

5

65

8

73

1

80

4

87

7

95

0

10

23

10

96

11

69

12

42

13

15

13

88

14

61

15

34

16

07

16

80

17

53

18

26

18

99

19

72

20

45

21

18

21

91

22

64

23

37

24

10

24

83

25

56

26

29



(i)

(ii)

(iii)

(iv)

-8000

-3000

2000

7000

11

05

20

93

13

41

75

21

62

57

29

83

39

37

10

41

11

45

12

49

13

53

14

57

15

61

16

65

17

69

18

73

19

77

20

81

21

85

22

89

23

93

24

97

26

01

Generator Transformer 01 - 220 kV side Currents

Serial number : 220 GEN TR1_IR A 653 220 GEN TR1_IY A

Sampling Rate : 220 GEN TR1_IB A 5000 220 GEN TR1_IN A

21

This is the Bus Section 1-3 fault at LVPS occurring after Unit 03, Unit 02 and Unit 01 going to FCB mode.

According to records, this fault has got cleared within 61ms by operating Busbar protection. Hence it

is clear that Phase B - N fault in the Bus Section 1-3 occurred as the last major event in the series and

it has nothing to do with the Generator Units at LVPS going into to FCB mode. The results of the

investigations on finding the exact reason for the Phase B - N fault in the Bus Section 1-3 was not

available at the time of generating this report.

Even if the Phase B - N fault did not occur in Bus Section 1-3 of LVPS, it is impossible to avoid the

activation of 𝑑𝑓

𝑑𝑡 in any of the units because as shown in Fig. 3.2d, the df/dt exceeded the set value of

2.0 Hz/s threshold in the ROCOF relay, which took place during the three phase to ground fault at

Kerawalapitiya GSS which took 154 ms to clear.

However, Bus 2 remained stable until the system collapsed.

Fig. 3.2m

It was further noted that no auxiliary power supplying mechanism, which is as high as 10% of the

installed capacity, was available at the LVPS to accommodate the controlled shut down process of the

units. If adequate auxiliary power was available, the units could have subsequently been restarted

without waiting for 3, 4 days with the need to replace the mechanical protection devices which got

activated under uncontrolled shutdown. Further it could have avoided eliminated the huge financial

losses incurred due to unavailability of LVPS to grid for the said long period counted in days.

22

3.2.6 Fast Cut Back (FCB) operation at the LVPS

FCB operation is implemented in DCS control to put the generators into house load operation in case

generator transformer circuit breakers of Units have tripped due to external faults. In the house load

operation, generators will run at 3000 rpm powering auxiliaries in the plant. But the house load

operation is not recommended to maintain more than 30 minutes. The main objective of FCB

operation is to keep the generators running with auxiliaries until the grid is powered up. If the grid

power is restored in time, electricity generation could be resumed. The FCB operation will be

activated if one of the following three condition is met

1. df/dt relay activation

2. forward power protection activation

3. 35MW/s load drop of the unit is measured on a given condition i.e. the unit should be

operated more than 150MW and a rapid unit power drop to an operating point less than

80MW in 2s condition is monitored. (Subjected to operator enables the FCB function from

the DCS). This condition is enabled only for the Generator Unit 02.

3.2.7 Time sequence analysis of Over Speed Protection Control (OPC) at the LVPS

OPC comes into operation while the Generator Units are synchronized to the grid the generator

circuits breakers are ON. Once the over speed protection control is activated, high pressure regulating

valves and low pressure regulation valves will be closed, making a drastic reduction in the generated

power leaving the room for system to reduce the frequency. OPC will be reset to an assigned dead

band and currently it is set to 3000 rpm for all the units. The present OPC setting at the LVPP units is

as shown in the Table 3.2e.

Table 3.2e: Present OPC settings at the LVPS

Setting Unit 1 Unit 2 Unit 3

OPC 3100 rpm 3078 rpm 3110 rpm

RESET OPC 3000 rpm 3000 rpm 3000 rpm

DEAD BAND 100 rpm 78 rpm 110 rpm

OPC NO YES NO

The OPC triggering can be verified by plotting the Generator Current against time (sample number)

and frequency against time (sample number) in the same plot. It is understood that the frequency

cannot be calculated at a rate faster than 100 Hz by using the Voltage Zero Crossing method on a 50

Hz voltage or current signal. Since the frequency at LVPS is monitored using a 20 Hz sampled recorder

and since that is not sufficient for this kind of time sequence analysis of the OPC operation, the author

23

of the report has written a multilevel frequency evaluation algorithm and used on the voltage data

recorded at BEN6000 fault recorder at LVPS to find the Generator frequency of each Unit. The

corresponding findings are shown in Figs. 3.2n, 3.2o and 3.2p for the Generator Unit 01, Unit 02 and

Unit 03 respectively. The algorithm stabilizes within the first two cycles and frequency data is accurate

beyond that point.

Generator Unit 01: It is observed in Fig. 3.2n(a) that there are two peaks in the generator frequency

of Unit 01. The first generator frequency peak has a magnitude of 50.87 Hz (3052.2 rpm) and occurs

during the 3 Phase to Ground fault at Kerawalapitiya GSS. The second generator frequency peak has

a magnitude of 53.33 Hz (3199 rpm) and as can be seen in the Fig. 3.2n(a), several other incidents also

happen there. Therefore, the last 297 data points of Fig. 3.2n(a) is zoomed and shown on Fig. 3.2n(b).

Fig. 3.2n(a): Variation of Generator frequency and current over sample number

(derived using voltage measurements sampled at 200 µs) of Unit 01

As can be seen in Fig. 3.2 n(b), the incident (i) occurs first and it is the feeding of the fault current in

the Bus Section 1/3. Next is the incident (ii), where the generator frequency starts increasing. In this

context, such a speed increase can only happen when the generator is disconnected from the network.

Such a scenario is possible if the Unit 01 enters FCB. The third incident is hitting the OPC trigger point

at 3100 rpm (51.67 Hz). However, the Unit 01 is already in FCB and therefore OPC does not operate.

Therefore, the second generator frequency peak has no resultant effect to the system.

0 500 1000 1500 2000 2500 3000-100

-80

-60

-40

-20

0

20

40

60

Generator 1 frequency (Hz)

Generator 1 current (A/500)

24

Fig. 3.2 n(b): Zoomed Generator Unit 01 current and Generator Frequency during the last 2400 to

2697 datapoints.

Generator Unit 02: It is observed in Fig. 3.2o(a) that there are two peaks in the generator frequency

of Unit 02. The first peak in the Generator Unit 02 frequency has an amplitude of 50.92 Hz (3055.2

rpm) during the 3 Phase to Ground fault at Kerawalapitiya GSS, and the second peak with amplitude

51.3 Hz (3078 rpm) occurs when the unit is apparently in FCB. Since the sequence of occurrence of

the second peak in the Generator Unit 02 frequency is not clear, the last 2000 to 2697 data points are

zoomed as shown in Fig. 3.2o(b).

As can be seen in Fig. 3.2o(b), there is a rise in generator current at (i) which corresponds to taking the

load partially from the Generator Unit 03 going into FCB mode. Subsequently there is a drop in system

frequency which is seen in (ii). According to Fig. 3.2o(b), at (iii), the Generator Frequency starts rising

as opposed to system frequency fall. Therefore, this point must correspond to the Unit 02 going to

FCB. However, it is not visible until (iv), where it can clearly be seen that the Unit 02 is in FCB. It is

interesting to note that the Unit 02 OPC trigger level of 3078 rpm (51.3 Hz) is met only at point (v),

which is after FCB. Therefore, even though OPC is noted in the Table 3.2e in the data provided by the

LVPS, it happens after FCB.

25

Fig. 3.2o(a): Variation of Generator frequency and current over sample number


Fig. 3.2 o(b): Zoomed Generator Unit 02 Current and Generator Frequency during the last 2000 to

2697 datapoints.

0 500 1000 1500 2000 2500 3000-60

-40

-20

0

20

40

60



26

It is observed in Fig. 3.2p (a) also that there are two peaks in the generator frequency of Unit 03 as

50.92 Hz (3055.2 rpm) during the 3 Phase to Ground fault at Kerawalapitiya GSS and 51.52 Hz (3091.2

rpm) when the unit is in FCB. Unit 03 in FCB can be verified by comparing the frequency curve with

the current curve and also the zoomed version of the same from the datapoint 2000 to 2697.

Fig.3.2p(a): Variation of Generator frequency and current over sample number


In Fig. 3.2 p(b), it can be seen that the Generator Unit 03 enters FCB at (i). However, there is a delay

to start rising the Generator Frequency as seen in (ii).

0 500 1000 1500 2000 2500 3000-60

-40

-20

0

20

40

60



27

Fig. 3.2 p(b): Zoomed Generator Unit 03 Current and Generator Frequency during the last 2000 to

2697 datapoints.

Hence it is verified that even if the speed logic conditions are satisfied for the OPC of a particular

Generator Unit to trigger, since it happens after the Unit entering to FCB mode with the generator

circuit breakers open, OPC cannot come into effect.

28

3.3. Observations on the LVPS behavior in this incident

1. Generator Unit 03, Unit 02 and Unit 01 respectively went into Fast Cut Back (FCB) mode

in that sequential order after detecting the system frequency having df/dt above their

threshold limits of 2 Hz/s. This high df/dt has been an external event to LVPS.

2. The B-N Phase to Ground fault in Bus Section 1-3 had occurred even after the third

Generator (Generator Unit 01) going in to FCB mode. Hence it has no connection to the

Generator Units going into FCB.

3. Current df/dt protection scheme appears to be not the most appropriate scheme which

should have been there for the LVPP.

4. Had the df/dt thresholds of the Generator Units were set unequal, say 2 Hz/s, 3 Hz/s, 4

Hz/s a total failure could have been avoided.

5. The df/dt control logic seems to have been adopted without adequate justification on

the trade-off between the machine safety and the risk of collapsing the entire system.

6. Even after going to FCB, had there been a means of auxiliary power which is in the range

of 90 MW, a controlled shutdown could have been exercised, which could have avoided

3-4 days long, expensive startup procedures.

3.4 Root Cause of the incident related to LVPS

System frequency rise at a rate above the threshold setting of 2 Hz/s caused the Generator Units at

LVPS to go into FCB mode and disconnected from the network, which made a huge loss of generation

to the network, leading under frequency tripping of other generator units in the network, despite

some automatic load shedding.

NOTE:

If not for the current waveforms in Figs. 3.2a, 3.2b, 3.2c, 3.2i, 3.2j and 3.2k obtained using the

BEN6000 fault recorder at LVPS with sampling period 200 µs, the conclusions derived just by

observing the event plots in the respective Unit DCS reproduced in Figs 3.2f, 3.2g and 3.2g, which

are with sampling periods 105 ms, 200 ms and 105 ms respectively, could have lead to serious errors

misleading the reader.

29

4. System Control Centre / CEB.

4.1 Relevant findings of the committee during the visit to the System Control Office (visited on the

22nd August 2020)

The DGM, Chief Engineer and the Electrical Engineer at the System Control/CEB presented system

failure they observed at the System Control as shown in Fig. 4.1a, where it is clearly seen that the bus

bar voltages at the fault in Kerawalapitiya Grid Substation and the Kotugoda Grid Substation both

drops to 0V from 220 kV.

System Control Engineers presented the procedure followed to restore the system subsequent to the

system power failure. The same information titled “sequence of system restoration attempts” have

been submitted to the committee. Restoration has started in Mahaweli Complex, Laxapana Complex

and Samanalawewa Complex simultaneously. Out of them, the Samanalawewa Complex feeds mostly

the rural loads. After a couple of over frequency failures, the Samanalawewa Complex stabilizes at

15:37. Hence, in 3 hours and 7 minutes, part of the Sri Lankan power network becomes live since the

power failure at 12:30. After several attempts, at 18:08 Samanalawewa System & Kukule System

synchronized via Ambalangoda- New Galle 132kV circuit 01.

However, the Mahaweli Complex and the Laxapana Complex feed urban and industry loads. Industry

loads are typically large loads and therefore when energizing the network from a blackout, there is a

higher probability to mismatch the generation to the dispatch. As a result, there can be more over

frequency failures in stabilizing the Mahaweli Complex and the Laxapana Complex. At 18:46 Mahaweli

Complex is stabilized and syncrnoized to Samanalawewa Complex via New Laxapana- Polpitiya 132kV

circuit 1. Hence the transmission network has come back to operation fully 6 hours and 16 minutes

from the system failure.

Fig. 4.1a

0

10

20

30

40

50

60

0

50

100

150

200

250

12

:30

:25

.88

8

12

:30

:26

.13

8

12

:30

:26

.38

8

12

:30

:26

.63

8

12

:30

:26

.88

8

12

:30

:27

.13

8

12

:30

:27

.38

8

12

:30

:27

.63

8

12

:30

:27

.88

8

12

:30

:28

.13

8

12

:30

:28

.38

8

12

:30

:28

.63

8

12

:30

:28

.88

8

12

:30

:29

.13

8

12

:30

:29

.38

8

12

:30

:29

.63

8

12

:30

:29

.88

8

12

:30

:30

.13

8

12

:30

:30

.38

8

12

:30

:30

.63

8

12

:30

:30

.88

8

12

:30

:31

.13

8

12

:30

:31

.38

8

12

:30

:31

.63

8

12

:30

:31

.88

8

12

:30

:32

.13

8

12

:30

:32

.38

8

12

:30

:32

.63

8

12

:30

:32

.88

8

12

:30

:33

.13

8

12

:30

:33

.38

8

12

:30

:33

.63

8

12

:30

:33

.88

8

12

:30

:34

.13

8

12

:30

:34

.38

8

Kerawalap 220 kV Kotugoda 220 kV Kolonnawa 132 kV Frequency

53.729 Hz50.453 Hz

30

4.2 Analysis of the observations made during the visit to System Control Centre

The restoration attempts in the Mahaweli Complex are taken for analysis because it contains unusual

complications which were not seen in the other two complexes.

Restoration Attempt 01: The System Control had initiated the restoration with the Victoria Power

Station Unit 02, which was ready at that time (12:57 pm). They had restored up to Biyagama Grid

Substation in 7 minutes, and the Unit 02 had tripped due to excitation stage II failure, which is an

internal fault in the generator, after 6 minutes at 13:10. This could be due to load turning OFF in the

33 kV feeder taken for restoration.

Restoration Attempt 02: The second attempt had been with the Victoria Power Station Unit 01 at

13:21 which had been dispatching for 3 minutes and failed at 13:27 due to over frequency tripping,

which could be due to load tripping.

Restoration Attempt 03: The third attempt also with the Victoria Power Station Unit 01 at 13:33 which

also had failed at 13:36 due to excitation stage II failure, which is an internal fault in the generator, a

similar reason as in the first attempt, 3 minutes after restoring.

Restoration Attempt 04: Since there are lesser number of circuit breakers to Biyagama Grid

Substation, the fourth attempt had been from the Kothmale Power Station Unit 03 at 13:36, which

had energized Biyagama Grid Substation in 3 minutes at 13:41 and subsequently the Kotugoda,

Aniyakanda and Sapugaskanda Grid Substations respectively by 14:05. However, Kothmale Power

Station Unit 03 had tripped due to over frequency after delivering 30 MW for 12 minutes. At this point

a system frequency oscillating at a lower frequency than 50 Hz had been observed.

Restoration Attempt 05: The fifth attempt had been started from Kothmale Power Station Unit 01

and subsequently added Kothmale Power Station Unit 02 but again tripped due to over frequency

after about one hour of dispatching power while delivering 85 MW with the system frequency

oscillating at a lower variable frequency of approximately 8 Hz. The associated frequency variation

against time is shown in Fig. 4.2b.

Restoration Attempt 06: The sixth attempt had been from Kothmale Power Station Unit 03, tripped

the generator this time after 10 minutes delivering 23 MW due to over frequency.

Restoration Attempt 07: The seventh attempt had been successful, with the Kothmale Power Station

Units 01 and 02 with the Unit 02 generator governor on manual mode, i.e., with the frequency control

loop being open.

31

Fig. 4.2b

The failures in the restoration of the first and the third attempts were solely due to internal faults of

the units 01 and 02 of the Victoria Power Station.

The failures in the restoration of the second and the sixth attempts can be attributed to over frequency

caused by one or many of the large loads in the selected 33 kV feeders got tripped or a large solar PV

system coming into operation, which is reflected at the generator as an over frequency.

The failures of the fourth and the fifth restoration attempts could be attributed to over frequency due

to generator electrical frequency oscillating at a low frequency in the range of 10 Hz. This has

happened with two generators in the same busbar. Hence it is an electrically tight coupled situation

caused by reduced damping in the electrical governor of one or both of the generators. In this case,

it must be Kothmale Power Station Unit 02 generator governor, because in the 5th attempt, Units had

tripped when Unit 02 added and the 7th attempt was successful when the Unit 02 generator governor

was put on manual mode., i.e., frequency control loop being open. Improving the damping in the Unit

02 generator governor of the Kothmale Power Station may be needed.

It is further observed that the SCC has not followed the Restoration Manual in restoring the

transmission network. The explanations received was that the generators at the Victoria Power

Station were ready to connect while the others were not. From the surface, the long delay over six

hours to restore the transmission network can be attributed to the fact that SCC has not followed the

restoration manual. However, there is no guarantee that following the Restoration Manual would

ensure quicker restoration, because the restoration attempts of the Colombo City via the Mahaweli

Complex have failed in the 1st, 3rd, 4th and the 5th attempts due to internal faults in the stations, which

44

45

46

47

48

49

50

51

52

53

54

55

8/1

7/2

02

0 3

:32

:00

PM

8/1

7/2

02

0 3

:32

:04

PM

8/1

7/2

02

0 3

:32

:08

PM

8/1

7/2

02

0 3

:32

:12

PM

8/1

7/2

02

0 3

:32

:16

PM

8/1

7/2

02

0 3

:32

:20

PM

8/1

7/2

02

0 3

:32

:24

PM

8/1

7/2

02

0 3

:32

:28

PM

8/1

7/2

02

0 3

:32

:32

PM

8/1

7/2

02

0 3

:32

:36

PM

8/1

7/2

02

0 3

:32

:40

PM

8/1

7/2

02

0 3

:32

:44

PM

8/1

7/2

02

0 3

:32

:48

PM

8/1

7/2

02

0 3

:32

:52

PM

8/1

7/2

02

0 3

:32

:56

PM

8/1

7/2

02

0 3

:33

:00

PM

8/1

7/2

02

0 3

:33

:04

PM

8/1

7/2

02

0 3

:33

:08

PM

8/1

7/2

02

0 3

:33

:12

PM

8/1

7/2

02

0 3

:33

:16

PM

8/1

7/2

02

0 3

:33

:20

PM

8/1

7/2

02

0 3

:33

:24

PM

8/1

7/2

02

0 3

:33

:28

PM

8/1

7/2

02

0 3

:33

:32

PM

8/1

7/2

02

0 3

:33

:36

PM

8/1

7/2

02

0 3

:33

:40

PM

8/1

7/2

02

0 3

:33

:44

PM

8/1

7/2

02

0 3

:33

:48

PM

8/1

7/2

02

0 3

:33

:52

PM

Freq

uen

cy (

Hz)

Time

Frequency oscillations during Restoration Attempt 05

32

are beyond the control of the SCC and they are not visible until the units are connected. The

Restoration Manual also assumes that the machines are healthy and can be started and connected

reliably when a need arises. Probably it is due to this reason, the staff of the SCC has lack of confidence

Restoration Manual. In addition, it is a fact that the accuracy of the Restoration Manual can not be

tested practically. Therefore, it is vital that such a manual is prepared after rigorous dynamic studies

of the system using accurate dynamic models of the network components in the system, taking all

possible practical scenarios into consideration.

4.3 Observations on the CCTV records at the Systems Control Center

The CCTV records during the event on the 17th August 2020 at the CEB Systems Control Center show

the activities of the two engineers who were on duty during the time of fault. It is observed that both

the engineers were paying attention on laptop computers in front of them and it could be seen that

one of them browsing the Internet while the other one accessing the Facebook. It is questionable as

to whether the code of ethics in line with standard professional practicing would accommodate such

a conduct, related to a professionally challenging and extremely sensitive task where urgent attention

cannot be compromised. Even though they can’t avoid a blackout of this nature, there can be some

alarms/signals which could be attended and prevent any possible failures in the system if they keep

eye on the system.

33

5. Conclusions

1. The three phase to ground fault at Busbar No. 02 of the Kerawalapitya GSS on the 17th August

2020 at 12.30.27.018 hrs. has caused the island wide power failure. This is due to incorrect

operation of the isolator and earth switches of the bus coupler bay during a routine maintenance

by the Electrical Superintendent, in charge of the Kerawalapitiya GSS. It is also noticed that the

said maintenance work has been carried out by the electrical superintendent and his staff without

being supervised by his superiors. It is observed that has been no evidence of clear instructions

provided by Engineers of the Transmission Operation and Maintenance division who are

responsible for maintenance activities at Kerawalapitiya GSS to the ES in charge and his staff at

Kerawalapitiya GSS.

2. Three Phase to earth fault occurred during a routine maintenance/inspection task at the

Kerawalapitiya GSS has caused tripping of all transmission lines and generator transformer circuit

breakers connected to this GSS by the operation of busbar protection. By-passing the general

interlocking mechanism between the Bus Coupler Circuit Breaker and the Busbar 2 of the

Kerawalapitiya GSS without considering the potential risk of human error in carrying out the said

routine maintenance/inspection task is the root cause for this incident.

3. Due to the three phase to ground fault at Kerawalapitiya GSS, due to relatively longer time taken

to clear the said fault and due to lack of system inertia, the rate of change of system frequency

has increased beyond the current settings of the generator df/dt - Rate Of Change of Frequency

(ROCOF) protection relay of the LVPS. As a result, the generator transformer circuit breakers of all

three units of the LVPS have tripped in the chronological order Unit 03, Unit 02 and Unit 01,

making the LVPS unavailable to the national grid of Sri Lanka, which triggered cascaded under

frequency tripping of other generators in the network, despite automatic load shedding observed

in the post fault system frequency plot.

4. The SCC has not adhered to the restoration procedure stipulated in the Restoration Manual to

restore the power to the Colombo City. However, it cannot be concluded that the restoration got

delayed over six hours solely due to not following the Restoration Manual because out of 6

unsuccessful attempts to restore Colombo City via the Mahaweli Complex, 4 attempts failed

because of internal problems in the generator units, which are beyond the control of SCC and will

only popup once they are attempted.

5. The first option given in the Restoration Manual to restore Colombo City, i.e., using GT 7 machine

in Kelanithissa Power Station to energize the Kolonnawa GSS after getting auxiliary supply from

frame V Gas Turbines, perhaps the fastest option, has not been available due to a long overdue

repair. Because of its nonavailability without an alternative, has also contributed significantly to

the long delay in the restoration.

6. The generator units LVPS had been available as Unit 01 until 12:42, Unit 03 until 13:33 and Unit

02 until 14:51, where there was no energized transmission network to feed in. Had there been a

faster transmission network restoration, say via Gas Turbines at Kelanithissa Power Station, the

system could have been restored about 4 hours earlier.

7. The governors of the hydro-power machines, particularly in the Kotmale Power Station, in manual

mode during the system restoration had enabled a stable ramping than the automatic governor

control at the system restoration.

8. No load prediction tools have been used to predict the feeder loads at the time of loading the

feeders. This has led to mismatch between generation and loads, which caused over frequency

and under frequency tripping (ex: the 2nd and the 6th restoration attempts of via the Mahaweli

Complex).

34

6. Recommendations

1. Establishing a standard compliant, systematic, foolproof, safe procedures and maintenance

protocols during operation and maintenance in CEB is highly recommended. The

implementation of these procedures will have to be continuously monitored and supervised by

properly qualified, professionally trained, knowledgeable, experienced and skilled personnel. It

is further proposed a performance evaluating annual appraisal system which will help to

improve the above attributes of the CEB staff.

2. Establishing a risk management mechanism in order to determine the proper mix of preventive

measures, mitigation levels, shift or retention of risks and consequent level of robustness of

Operations & Maintenance protocols that would indicate the positive impact on the overall

system.

3. Establishing the Long Term Generation Expansion Plan in line with the current government

policy, considering the future requirements of the Sri Lankan power system in view of the

optimum generation mix based improved system stability, reliability, energy security and least

cost to the society.

4. Conducting a comprehensive review by CEB on the settings of the protection relays in the entire

network as it is evident that if the Kerawalapitiya Busbar protection 2 had operated quicker than

154 ms, the lack of inertia in the system would have passed unnoticed and the system could

have been prevented from going into a total blackout. It is further recommended to review the

existing protection strategy for frequency instability criteria used for df/dt in the LVPS.

5. Synchronizing all clocks in the national grid with the GPS clock is recommended so that the data

logged is consistent and easy to follow.

6. Reinstating Colombo City restoration Option 1 as per the current restoration manual, i.e., via Kalanithissa Power Station. Failing that, it is recommended to install and commission 70 to 100 MW gas turbines with frequency control, black start, ability to operate at leading and lagging power factors, stability at small loads, line charging for capacitive loads, features.

7. Studying and identifying the best mode of governor controls and settings to facilitate fast system restoration. It is evident that an improved damping in the frequency control loop in the generator unit 2 of the Kothmale Power Station could have saved a significant amount of time to restore the system.

8. Investigating better means of using past daily loading records of the feeders in the distribution network to predict more accurate load demands as this would drastically reduce the time to restore the complete transmission network.

9. Ensure the availability of an updated system restoration manual verified through accurate dynamic simulation studies of the system, so that the system can be restored reliably when it is used as a guide.

10. Staff involved in the control centres of the power plants and as well as those in the SCC, be receiving continuous professional training to become experts in their duties and to be disciplined to follow and handle emergency situations at the highest benchmarked skill levels.

11. Enforcing strict discipline on the staff of SCC while at work, recognizing the gravity of the correct execution of their duties.

35

Signed by

………………………………………

Dr. Lilantha Samaranayake Department of Electrical & Electronic Engineering Faculty of Engineering University of Peradeniya

final report version ii on the total power system failure

Documents