DGA Consulting International Comparison of Major Blackouts and Restorat

INTERNATIONAL
COMPARISONOFMAJOR
BLACKOUTSAND
RESTORATION
AEMCReliabilityPanel
Authored by:
Date of issue:
Version:
DGA Consulting
PO Box 1061
Hunters Hill, NSW 2110
AUSTRALIA
5/05/2016
3.0
DocumentControl
CustomerDetails
CustomerName:
CustomerAddress:
ContactPerson:
POReference:
AboutthisDocument
Title:
DateofIssue:
TermsandConditions:
Preparedby:
Approvedby:
RevNo. Date
1.2
13/3/16
1.3
17/3/16
2.0
29/3/16
2.1
30/3/16
2.2
29/4/16
3.0
5/5/16
AEMCReliabilityPanel
Level6,201ElizabethSt
Sydney,NSW2000
JulianEggleston
mailto:julian.eggleston@aemc.gov.au
p:0282967800
Internationalcomparisonofmajorblackoutsand
restoration
05/05/2016
DGAConsultingareengagedunderagreedTerms
andConditionswithAEMC
P.JeffreyPalermo
DonBonnitcha
ReasonforIssue
Firstdraft
Firstdraftrevised
SeconddraftwithTask3
Fig1andrevisedtext
Addeddiscussionofinertiaand
faultcurrent,andminor
corrections
Clarificationsandcorrecttypos
Updatedby
PJP
PJP
PJP
PJP
PJP
Verifiedby
DB
DB
DB
DB
DB
PJP
DB
PO Box 1061 Hunters Hill, NSW 2110 Tel: +61 2 8005 7471
DGA Consulting Pty Limited ABN 22 600 478 859
www.dga-consulting.com.au
Contents
Projectoverview...................................................................................................................................1
UtilityandregulatorystructureintheAustralianNEM..............................................................1
Developmentsin2015regardingSRASandblackstart................................................................1
Task1Internationalcomparisonofmajorblackouts...................................................................3
Contextforcomparison...................................................................................................................3
Internationalcomparison.................................................................................................................4
Summary............................................................................................................................................8
2003EasternUS.................................................................................................................................8
Thesituationbeforetheblackout..............................................................................................10
Theblackout.................................................................................................................................11
Restoration....................................................................................................................................16
Lessonslearned............................................................................................................................22
2013Sarawak...................................................................................................................................23
Theblackout.................................................................................................................................24
Restoration....................................................................................................................................25
Lessonslearned............................................................................................................................26
2008Oahu,Hawaii.........................................................................................................................26
Theblackout.................................................................................................................................28
Restoration....................................................................................................................................28
Lessonslearned............................................................................................................................29
2003Italy..........................................................................................................................................30
Theblackout.................................................................................................................................31
Restoration....................................................................................................................................33
Lessonslearned............................................................................................................................40
2011SanDiego,US.........................................................................................................................41
PO Box 1061 Hunters Hill, NSW 2110 Tel: +61 2 8005 7471
DGA Consulting Pty Limited ABN 22 600 478 859
www.dga-consulting.com.au
DGAConsulting
Theblackout.................................................................................................................................42
Restoration....................................................................................................................................45
Lessonslearned............................................................................................................................46
Task2Internationalcomparisonofregulatoryarrangementstopreventormitigate
blackoutsincludingrestoration..................................................................................................47
Contextforcomparison.................................................................................................................47
PJM,US............................................................................................................................................50
Specificsystemrestorationvariables........................................................................................51
Underlyingsystemconditionassumptions.............................................................................52
SouthAfrica.....................................................................................................................................53
Italy...................................................................................................................................................53
ERCOT,US......................................................................................................................................54
Ireland..............................................................................................................................................56
Restoringpriorityloads..............................................................................................................56
Underlyingsystemconditionassumptions.............................................................................56
Task3Changesanticipatedforblackstartplanninginsystemswithveryhighlevelsof
renewablegeneration...................................................................................................................57
Natureofrenewablegenerationaffectingblackoutsandblackstart.....................................57
Impactonblackouts.......................................................................................................................58
Systeminertia...............................................................................................................................58
Ramprates(rateofchangeoffrequency)................................................................................59
Underfrequencyandundervoltageresponse.......................................................................60
Coincidentfrequencyreaction...................................................................................................61
Overvoltages...............................................................................................................................61
Restartcapability.........................................................................................................................62
Impactonrestoration.....................................................................................................................62
Possiblemitigationmeasures........................................................................................................63
Conventionalgeneration............................................................................................................63
AEMC Reliability Panel International comparison of major blackouts and restoration
Page ii
DGAConsulting
Increasedinertiaandfaultcurrents..........................................................................................63
UVLSandUFLSsystems................................................................................................................63
Invertercapabilityanddesign...................................................................................................64
SouthAustraliainterconnections..............................................................................................64
Energystorage.............................................................................................................................65
Conclusions.........................................................................................................................................67
Generalconclusionsregardingmajorblackouts........................................................................67
Generalconclusionsregardingblackstartrequirements.........................................................68
SpecificcommentsfortheAEMCandNEMsituation:.................................................................68
Figures
Figure1:ThreestagesofrestorationusedbyNEM.........................................................................4
Figure2:ReliabilityCoordinatorsintheaffectedarea................................................................10
Figure3:Situationat16:10:27emphasizingseparationborder..................................................13
Figure4:Situationat16:10:45emphasizingseparationborder..................................................14
Figure5:Situationat16:13showingthefinalseparatedarea....................................................14
Figure6:Theareathatwasultimatelyblackedout.....................................................................15
Figure7:Regionalloadandgenerationjustbeforeblackout.....................................................24
Figure8:Oahugeneration...............................................................................................................27
Figure9:Oahurestorationtimeline................................................................................................29
Figure10:SeparationsequenceofItalianinterconnections........................................................32
Figure11:Italyrestorationtimeline...............................................................................................34
Figure12:ItalyattheendofStage1restoration..........................................................................36
Figure15:SanDiegoareamapofevent........................................................................................42
Figure16:ThesouthernCaliforniasituationat15:28..................................................................43
Figure17:ThesouthernCaliforniasituationat15:32..................................................................44
Figure18:PJMs21transmissionzones..........................................................................................51
Figure19:ERCOTareaofTexas........................................................................................................55
Tables
Table1:Internationalcomparisonofmajorsupplydisruptions..................................................5
Page iii
DGAConsulting
Table2:SummaryofHECOgeneration..........................................................................................28
Table3:Italianimportsat03:0028September2003.....................................................................31
Table4:SanDiegoCountypreeventsupplyanddemand........................................................42
Table5:SanDiegoblackoutstatistics.............................................................................................45
Table6:SanDiegoarealoadrestorationtimes.............................................................................46
Table7:SanDiegoareagenerationrestorationtimes..................................................................46
Table8:Internationalcomparisonofregulatoryarrangementstopreventorameliorate
blackouts................................................................................................................................48
Page iv
DGAConsulting
PROJECT OVERVIEW
Utility and regulatory structure in the Australian NEM
TheAustralianEnergyMarketCommission(AEMC)andtheReliabilityPanelarepartofthe
utilityregulatorystructureoftheAustralianNationalElectricityMarket(NEM).TheNEMis
awholesalemarketforsupplyingelectricitytoretailersandendusersinQueensland,New
SouthWales,theAustralianCapitalTerritory,Victoria,SouthAustraliaandTasmania.
Operationsarebasedinfiveinterconnectedregionsthatlargelyfollowstateboundaries.
TheAEMCoperateswithinabroadermarketgovernancestructurealongsidetheAustralian
EnergyMarketOperator(AEMO)andtheAustralianEnergyRegulator(AER).TheAEMC
determinesthepolicyenvironmentandgovernancestructuresthatshapeAustralias
developingenergymarketsandsetstheoperatingrequirementsandobligationsofmarket
participants.TheAEMOmanagestheNEMandgasmarkets.
AmongtheservicestheAEMOmanagesisSystemRestartAncillaryServices(SRAS).The
objectiveofSRASistoprovidereasonableassurancethatthesystemcanberestarted
followingaregionalblackout.TheAEMOistoprocuretheleastcostcombinationofSRAS
submissionsthatmeettheSystemRestartStandard(SRS).
TheSRSisdeterminedbytheReliabilityPaneltomeettherequirementsoftheNational
ElectricityRules(NER).Specifically,theSRSidentifiesthemaximumamountoftimethat
SRASareallowedenergizeaspecifiedgenerationleveltarget.TheSRSalsoidentifiesa
numberofotherparameters,includingthestrategic,geographic,technologyandfuel
diversityofSRAS,aswellastheprinciplesthatAEMOmustconsiderwhendevelopingthe
boundariesofelectricalsubnetworks.
Developments in 2015 regarding SRAS and black start

InApril2015theAEMCpublishedafinalrulechangethatamendedsomeofthe
arrangementsinrelationtotheprocurementofSRAS.On30June2015,theAEMCprovided
TermsofReferencetotheReliabilityPaneltoinitiateareviewoftheSRS.Amongother
things,thisrequiredthePaneltoundertakeareviewoftheSRStomeettherequirements
revisedinJuly2015followingafinalruledeterminationmadebytheAEMC.ThePanelis
requiredtocompleteitsReviewoftheStandardbyDecember2016.
Page 1 of 69
DGAConsulting
Aspartofitsresponse,theAEMCreleasedanIssuesPaperinNovember2015describingthe
issuesassociatedwithSRAS.Variousstakeholderssubmittedresponsestothisissuepaperin
December.
On29January2016theAEMCissuedarequestforproposals(RFP)asoneelementinthe
AEMCsandReliabilityPanelsresponsetotheserequirements.TheRFPsoliciteda
consultanttoprovideaninternationalcomparisonofmajorsupplydisruptionsinelectricity
systemsandregulatoryarrangementsforpowersystemrestoration.1TheRFPrequestedan
internationalcomparisonofeventswherethepowersystemhascollapsedtoablacksystem
conditioncommonlycalledablackout.2
TheAEMCselectedDGAConsultingPty.Limited(DGA)toperformthiswork.
TheAEMCsoughtaswideareviewaspossibleofdifferentrelevantinternationaleventsand
regulatoryarrangements.TheAEMCrequestedtwomainpartsofthework:
1. Aninternationalcomparisonofmajorblackouts;and
2. Aninternationalreviewofregulatoryarrangementstopreventor
mitigatesuchoutagesincludingrestoration.
ThisreportaddressesthesetwomainpartsinTasks1and2,below.
TheAEMCalsoelectedtohaveDGAperformanadditionaltaskproposedbyDGAtodescribe
blackstartplansintwoislandedsystems(IrelandandHawaii)wherelargeamountsof
intermittentgeneration(windandsolar)arebeinginstalled.Initialinquiriesrevealedthat
suchplanninginthosejurisdictionswasnotwelladvancedandthechapterwasadjustedto
explorethechallengesintroducedby,andpotentialmitigationapproachesfor,significant
levelsofintermittentgeneration.
1. AmajorsupplydisruptionisasdefinedinChapter10oftheNERtheunplannedabsenceofvoltageona
partofthetransmissionsystemaffectingoneormorepowerstationsandwhichleadstoalossofsupplyto
oneormoreloads.
2. BlacksystemisdefinedinChapter10oftheNERtheabsenceofvoltageonallorasignificantpartofthe
transmissionsystemorwithinaregionduringamajorsupplydisruptionaffectingasignificantnumberof
customerselectricitysupplydisruptionevent.
Page 2 of 69
DGAConsulting
TASK 1INTERNATIONAL COMPARISON OF MAJOR

BLACKOUTS
Theprocessandexperienceofrestoringthesystemafterablackoutisanimportantelement
oftherequiredwork.Whilethereareusuallypublishedreportsandotherinformation
regardingthecausesofspecificblackouts,thereisusuallylittlepublishedinformationabout
restoration.Inthereport,wehaverelied,inpart,oninformationprovidedinformallyby
variousutilityexpertsintheaffectedjurisdictions.
Context for comparison

TheAEMCisinterestedincomparinginternationalpracticesrelatedtoblackoutsandblack
startgeneration.SomeoftheissuesthattheAEMCisinterestedincomparingincludethe
causesoftheblackouts,howwellblackstartunitsperformed,thetimingofrestoration
steps,howinterconnectionswereusedinrestoration,howquicklyservicewasrestored,and
problemsthatoccurredduringrestoration.
Regardingtimingofrestoration,thecommonunderstandingintheNEMisathreestage
processasdepictedinFigure1.
Stage1theAEMOinitiallyassesseseventsandsystemconditions.Thefocus
isonrestoringgenerationandtransmissionnetworkstosupplyauxiliariesof
generatingunitsfromSRASbyenergizingalimitedtransmissionnetwork.
DistributionNetworkSystemProviders(DNSPs)reconnectloadasdirectedby
TransmissionSystemNetworkProviders(TNSPs)andtheAEMOtostabilize
thesystem.(Restorationmayincludelimitedsupplytosensitiveloadswhere
practical).
Stage2themajorityofthetransmissionnetworkisenergizedand
available/requiredgeneratingunitsarestarted.Thefocusisonrestoringthe
transmissionnetworkandrampingupgeneration.DNSPsprogressively
reconnectloadunderdirectionfromTNSPs,mainlytostabilizethesystem,
givingprioritytosensitiveloadswherepractical.
Stage3thedistributionnetworksenergizeanddistributedgeneration
progressivelyrestarts.Remainingdistributionloadsareprogressively
restored.TheendofStage3restoresnormaloperation.(Anynetwork
damageorrepairsrequiredmaydelayrestorationofsomeloads.)
Page 3 of 69
DGAConsulting
Figure 1: Three stages of restoration used by NEM
Figure1showstheobjectofthepresentSRSistorestoregenerationsuchthat40%ofthe
demandcouldbesuppliedwithin4hours.
International comparison
ThischapterdescribesfivemajorinternationalblackoutsasrequestedintheAEMCsRFP.
Fortunately,majorblackoutsareinfrequent,however,thismeansthereareonlyalimitedset
ofeventstochoosefrom.TheseblackoutswereselectedinconsultationwiththeAEMCfrom
alistofsuggestionsbyDGA.Thefiveselectedwere:
2003EasternUSA;
2013Sarawak,Malaysia;
2008Oahu,Hawaii;
2003Italy;and
2011SanDiego,California.
TheAEMCincludedspecificitemstodescribeeachblackoutintheirRFP.Eachoftheseis
summarizedinTable1,below.Inaddition,reportsectionsbelowdescribetheseeventsin
greaterdetail.
Page 4 of 69
DGAConsulting
Table 1: International comparison of major supply disruptions

Event
Characteristic
2003 Eastern US
2013 Sarawak
2008 Oahu, Hawaii
2003 Italy
2011 San Diego
General description
Major regional outage

(see page 8)
Total system outage

(see page 23)
Total island outage

(see page 26)
National outage
(see page 30)
Regional blackout
(see page 41)
NEM Stage 1
6 hours*
3 hours
5 hours
3 hours
N.A.
NEM Stage 2
10 hours
4 hours
9 hours
4 hours
4-6 hours*
NEM Stage 3
2 days for most

4 days for a few
8 hours
96% in 18 hours
99% in 15 hours
12 hours
Less than 30 minutes

for hydro and pumped
hydro
30 minutes
Less than 30 minutes
30 minutes
Only interconnections
were used
The physical extent of the event (capacity

or lost energy, etc.)
61,800 MW
1,600 MW
1,000 MW
35,000 MW
8,000 MW
Human errors, if any, that triggered or

helped propagate the event
Situation awareness
Switching error
None
Situation awareness
None
Protection equipment failures, if any, that

triggered or helped propagate the event
Software in EMS
Errors at Bakun dam
None
Multiple
None
The contribution of specific load or

generation types to the triggering of, or
restoring power following the event
None
None
None
None
None
Any unusual power system frequency,

voltage or stability issues that contributed to
the triggering or propagation of the event
Northern Ohio had

barely enough
reactive reserves
Synchronizing 100 km
275 kV lines at Bakun
Loss of generation
saw frequency fall
47 Hz (60 Hz normal)
Frequency dropped to
47 Hz when interconnections tripped
None
Time from the initial

system collapse to
restoring normal
operations
Time for restart services to come online to

begin system restoration
Page 5 of 75
* Estimated
The blackout was so extensive that some parts of the transmission system took longer to restore.
The Italian peninsula is very long, so the times are for the main northern portion of system.
Page 5 of 69
DGAConsulting
Event
Characteristic
2003 Eastern US
2013 Sarawak
2008 Oahu, Hawaii
2003 Italy
2011 San Diego
Any natural phenomena that contributed to

the cause and impact of the major supply
disruption event, such as weather or
seismic conditions
None
None
Lightning storm
None
None
The readiness of generation and supply

equipment and the linkages with restoration
time
Some non-black-start
generation had
problems
Normal
Normal
Normal, but only 8 of

31 TTHL units operated
Normal
The readiness of network assets and the

linkages to restoration time
Mostly normal, with

some damage in this
extensive blackout
Normal
Normal
Normal
Normal
Any equipment damage and the extent to

which this affected the restoration
None
None
A critical 138
kV line
Page 6 of 75
had physical damage
None
None
The extent to which power system topology

contributed to the propagation or
containment of the event, and the
subsequent restoration of the power system
Natural break points

limited propagation
The 100 km 275 kV

line to Bakun were a
problem restoring
None
Italy is very long

peninsula, it was hard
to energize Sicily
There are only two

major transmission
paths
Whether designated restart services were

the only source of restoration, or whether
supply from neighboring power systems
was utilized to assist in the restoration
process
Interconnections were
an important part of
restoration
No interconnections
No interconnections
Interconnections were
critical in restoration
The system was reenergized solely from

the interconnections
no black-start was
used
Did the system restart plan operate as

expected, in relation to timeframes and the
energizing process?
There were problems

across the areas
blacked out
Yes, except for

switching problems at
Bakun
Several failed restarts

between hours 3 to 5
Yes
Yes, but there was a

30-minute discussion
between TOs about
transmission during
re-energizing
Also, some pockets

remained with power
Page 6 of 69
DGAConsulting
Event
Characteristic
2003 Eastern US
2013 Sarawak
2008 Oahu, Hawaii
2003 Italy
2011 San Diego
Any particular element of the restart

process that presented specific hurdles in
system restoration (resources, generation,
transmission, distribution, load switching,
etc.)
The widespread
nature made situation
awareness a
challenge
Switching errors at
Bakun substation
None
Controlling voltages
None
The economic cost of the event, such as

value of lost load or impacts on gross
domestic/national product
6 to 10 billion USD
150 million USD
120 million USD
Any social impact attributable to the event
There was major

economic and social
disruption
Chaos during rush

hour in Kuching, the
capital
International
embarrassment
Occurred in early in
morning during
holiday festivities
Occurred in early
afternoon just before
rush hour
The cost of the restart services called upon

to restore the system following the event
None
None
None
None
None
Any key lessons learned or recommendations arising from investigations of the

event undertaken by affected parties or
third, parties including whether the event
was caused by an expected or unexpected
sequence of events
See text beginning at

page 22
Keep personnel
current with black-start
training
Nothing special
Coordinate UFLS
generation trip
settings
Nothing special
Page 7 of 75
Page 7 of 69
DGAConsulting
Summary
SomeofthehighlightsfromTable1arebrieflydiscussedherewithadditionaldetails
providedinthesectionsbelow.
Foreachoftheoutagesstudied,therestorationtimescomparabletoNEMstage1were36
hoursandroughlyconsistentwiththeSRS.Mostgenerationwasenergized(NEMstage2)in
46hours.Thenotableexceptionisthe2003USblackout.And,allcustomerloadwas
restored(NEMstage3)in1216hours,againwiththeUSexception.
Thesocialimpactvariedsomewhatamongtheblackouts.Hawaiiwasembarrassedbecause
PresidentObamaandhisfamilyweretherealongwiththeinternationalpresscorps.The
USblackoutaffectedalargeportionofthecountryforseveraldaysandhadamajor
economicimpact.Incontrast,theItalianblackoutoccurredduringtheeveningofanational
holidaywheneconomicactivitywaslowwithshopsandbusinessesclosedfortheday.
TheestimatedcostoftheblackoutsrangedfromUSD610billionto120million.The
approximateUSDcostperMWhwere7,500(US2003),2,500(SanDiego),and900(Italy2003).
Blackstartgeneratorsperformedasexpectedbyenergizinginabout30minutes.Thetrip
tohouseloadgeneration(TTHL)inItalyperformedpoorlywithonly8of31unitsstarting.
(TheItaliansystemoperatorhassinceinstitutedarigoroustestingregimefortheseunits.)
Allthesystemoperatorsusedtheirinterconnectionsveryearlyinrestoringtheirsystems,
andSanDiegorestartedwithoutactivatinganyofitsblackstartresources.
Allthesystemshadsomeelectricalislandsremaininservicethroughouttheblackout.And,
inallcasesexceptHawaii,thetransmissionsystemwasintactfollowingtheinitialevent.
2003 Eastern US
At16:10on14August2003oneoftheworldsworstblackoutsoccurredintheeasternUS.
Theoutageaffectedanestimated50millionpeopleand61,800megawatts(MW)ofelectric
loadinthestatesofOhio,Michigan,Pennsylvania,NewYork,Vermont,Massachusetts,
Connecticut,NewJerseyandtheCanadianprovinceofOntario.Powerwasnotrestoredfor
4daysinsomepartsoftheUnitedStates.PartsofOntariosufferedrollingblackoutsfor
morethanaweekbeforepowerwasfullyrestored.EstimatesoftotalcostsintheUnited
Page 8 of 69
DGAConsulting
Statesrangebetween$4billionand$10billion(USD).3InCanada,thenationalgross
domesticproductwasdown0.7%inAugust,therewasanetlossof18.9millionworkhours
andmanufacturingshipmentsinOntarioweredown$2.3billion(CDN).4
ThecascadingblackoutbeganinOhio,spreadeastward,andcausedsuchwidespread
outagesforthreeprincipalreasons:
1. Thelossofakey345kVlineinOhio,followingthelossofothertransmission
linesandweakvoltageswithinOhiothattriggeredmanysubsequentline
trips.
2. Manyofthekeylineswhichtrippedbetween16:05and16:10,operatedon
zone3impedancerelays(orzone2settooperatelikezone3)thatresponded
tooverloadsratherthantruefaultsonthegrid.5Thespeedatwhichthey
trippedspreadthereachandacceleratedthespreadofthecascadebeyond
thenorthernOhioarea.
3. Relayprotectionsettingsforthetransmissionlines,generatorsandunder
frequencyloadsheddinginthenortheastUSwerenotentirelyappropriate
andwerecertainlynotcoordinatedandintegratedtoreducethelikelihood
andconsequencesofacascade.
ComparedwithotherblackoutsnewcausalfeaturesoftheAugust14blackoutinclude:
inadequateinterregionalvisibilityoverthepowersystem;dysfunctionofacontrolareas
SCADA/EMSsystem;andlackofadequatebackupcapabilitytothatsystem.
Thisblackoutalsooccurredbeforevariousregionalcontrolandmonitoringsystemswere
fullyoperationalattheMidwestIndependentSystemOperator(MISO).
3.
ICFConsultingICF(aUSconsultingfirm)estimatedthecoststobebetween$6.8and$10.3billion,The
EconomicCostoftheBlackout,AnissuepaperontheNortheasternBlackout,August14,2003,undated.
AndersonEconomicsGroupestimated$4.5to$8.3billion,PreliminaryEstimate:EconomicImpactofa1to3day
BlackoutInNortheastU.S.,August2003,undated
4. StatisticsCanada,GrossDomesticProductbyIndustry,August2003,CatalogueNo.15001;September2003
LabourForceSurvey;MonthlySurveyofManufacturing,August2003,CatalogueNo.31001.
5. Relaysarecommonlysetforzones.Zone1monitorstheimmediateequipmentandwilloperatequickly
usuallyinafewcycles.Zone2relaysmonitorfarther,usuallyasabackupforrelaysatheotherendofthe
line,theyoperatemoreslowlythanzone1.Zone3monitorsevenfartherandoperatesmoreslowlythan
zone2,thisarethelasthopetointerruptafaultifbothzones1and2fail.
Page 9 of 69
DGAConsulting
The situation before the blackout

ThegeneralareaaffectedbytheAugust2003blackoutisshowninFigure2.TheReliability
CoordinatorsinvolvedweretheMISO,theOntarioIndependentMarketOperator(IMO),the
NewYorkIndependentSystemOperator(NYISO),andthePJMInterconnection(PJM).
ReliabilityCoordinatorsprovidereliabilityoversightoverawideregion,theyprepare
reliabilityassessments,provideawideareaviewofreliability,andcoordinateemergency
operationsinrealtimeforoneormorecontrolareas.Theymayoperate,butdonot
participatein,wholesaleorretailmarketfunctions.
Figure 2: Reliability Coordinators in the affected area
Page 10 of 69
DGAConsulting
Thegeneralconditionsontheregionalsystemswerefairlynormalforasummerday:6
Loadswerehigh,butbelowpeaksummerconditionsabout90%ofprevious
peakloads;
Interregionaltransferswerehigh,butwiththenormalrange;
Voltageswerewithinacceptablelimits,withoperatorsactingtoboost
voltagesasisnormalduringasummerafternoon;
Frequencywastypicalforasummerafternoon;
Allsystemelementswerewithinnormalandcontingencylimits;and
Temperatureswerewarm,butgenerallyabout5CbelowAugusthigh
temperatures.
Therewereahandfulofgeneratorsonplannedmaintenance,however,inaregionalsystem
withanexpectedloadofabout300,000MW,thisisnormal.
The blackout
Therewerefourstagesoftheblackoutsinitiatingsequence:
1. Anormalafternoondegrades;
2. ThenorthernOhiosystemoperatorscomputerfailures;
3. ThreeOhio345kVtransmissionlinefailuresandmanyphonecalls;and
4. ThecollapseofthenorthernOhio138kVsystemandthelossofakey345kV
line.
Beginningat12:15,inaccurateinputdatarenderedMISOsstateestimator(asystem
monitoringtool)ineffective.About13:30a612MWgenerationunitinnorthernOhiotripped
(Eastlake5)andshutdownautomatically.7Losingthisunitdidnotputthegridintoan
unreliablestate,however.
6. MuchofthissectionofthisreportisbasedonU.S.CanadaPowerSystemOutageTaskForce,FinalReporton
theAugust14,2003BlackoutintheUnitedStatesandCanada:CausesandRecommendations,5April2004.
7. Eastlake5trippedofflineastheoperatorsoughttoincreasetheunitsreactivepoweroutput,theunits
protectionsystemdetectedthatvaroutputexceededtheunitsvarcapabilityandtrippedtheunitoffline.
ThelossoftheEastlake5unitdidnotputthegridintoanunreliablestatei.e.,thesystemwasstillableto
safelywithstandanothercontingency.However,losingtheunitrequirednorthernOhiotoimport
additionalpowertomakeupforthelossoftheunitsoutput(612MW),madevoltagemanagementin
northernOhiomorechallenging,andgavesystemoperatorslessflexibilityinoperatingtheirsystem.
Page 11 of 69
DGAConsulting
The initial events

Shortlyafter14:14,thealarmandloggingsysteminthenorthernOhiosystemoperators
controlroomfailedandwasnotrestoreduntilaftertheblackout.After15:05,some345kV
transmissionlinesincentralOhiobegantrippingoutbecausethelineswerecontacting
overgrowntreeswithinthelinesrightofwayareas.
About15:46whenthelocalsystemoperatorandMISOandneighboringutilitieshadbegunto
realizethatthenorthernOhiosystemwasinjeopardy,theonlywaythattheblackoutmight
havebeenavertedwouldhavebeentodropatleast1,500MWofloadaroundClevelandand
Akron.Nosucheffortwasmade,however,andby15:46itmayalreadyhavebeentoolate
foralargeloadshedtomakeanydifference.
Northern Ohio separates
After15:46,losingsomekey345kVlinesinnorthernOhiocausedtheunderlyingnetworkof
138kVlinestooverloadandtrip,leadinginturntothelossofacritical345kVlineat
16:06theeventthattriggeredtheinitialuncontrollable345kVcascadeportionofthe
blackoutsequence.Thelossofthis345kVlineshutdownthe345kVpathintonorthern
OhiofromeasternOhio.AlthoughtheareaaroundAkron(northwestOhio)wasalready
blackedoutduetoearlierevents,mostofnorthernOhioremainedinterconnected.Theloss
ofthisheavilyoverloadedkey345kVlineinstantlycreatedmajorandunsustainableloading
onlinesinadjacentareas,andthecascadespreadrapidlyaslinesandgeneratingunits
automaticallytrippedbyprotectiverelayactiontoavoidphysicaldamage.
Thesituationat16:10:27isshowninFigure3withOhiointhecenter,MichiganandOntario
tothenorth,andPennsylvaniaandNewYorktotheeast.8Theheavylineshowsthe
interfacewherelineshavetripped.Ascanbeseen,northernandsouthernOhioareno
longerinterconnected.
8. Figure3,Figure4,Figure5andFigure6aretakenfromtheFinalReportontheAugust14,2003Blackoutinthe
UnitedStatesandCanada:CausesandRecommendations,5April2004
Page 12 of 69
DGAConsulting
Figure 3: Situation at 16:10:27 emphasizing separation border
The separations rapidly spread

Duringthenext18secondsnumerousautomaticprotectiondevicesoperated.Thesituation
at16:10:45(Figure4)showshowtheseparationareaexpandedtoincludenorthernOhio,
southeasternMichigan,Ontario,andNewYork.
Page 13 of 69
DGAConsulting
Figure 4: Situation at 16:10:45 emphasizing separation border
LessthanthreeminuteslatertheblackoutwascompleteasshowninFigure5.
Figure 5: Situation at 16:13 showing the final separated area
Page 14 of 69
DGAConsulting
Thecascadebecamearacebetweenthepowersurgesandtherelays.Thelinesthattripped
firstweregenerallythelongerlineswithrelaysettingsusinglongerapparentimpedance
trippingzonesandnormaltimesettings.LinesinPennsylvania,thatarenothighly
integratedintotheelectricalnetwork,trippedquicklyandsplitthegridbetweenthesections
thatblackedoutandthosethatrecoveredwithoutfurtherpropagatingthecascade.
Thevastmajorityoftripoperationsonlinesalongtheblackoutboundariesbetween
PennsylvaniaandNewYork(forinstance)showedhighspeedrelaytargetsthatindicate
thatamassivepowersurgecausedeachlinetotrip.Totherelays,thispowersurgealtered
thevoltagesandcurrentsenoughthattheyappearedtobefaults.Thepowersurgewas
causedbypowerflowingtothoseareasthatweregenerationdeficient(northernOhioand
southeasternMichigan)orreboundingback.Theseflowsoccurredpurelybecauseofthe
physicsofpowerflowspowerflowsfromareaswithexcessgenerationflowingintoareas
thataregenerationdeficient.
Figure6showstheareaaffectedbytheblackout.PortionsofOhio,Michigan,Ontarioand
NewYorklostpower.
Figure 6: The area that was ultimately blacked-out
Page 15 of 69
DGAConsulting
Restoration
Asmightbeexpected,therewasacertainamountofchaosfollowingsuchahugeblackout.
Oncetheextentoftheareawasrecognized,restorationproceededinareasonablyorderly
way.Withsuchalargeaffectedareaandsomuchequipmentinvolved,however,therewere
numeroussetbacksandequipmentmisopertion,usuallyrelatedtohighvoltageswhen
transmissioncircuitswereenergizedbeforetherewaseithersufficientloadorgeneration
connected.
Itshouldbenotedthatthereweresmallpocketsofthesystemwithintheaffectedareathat
remainedwithpower.Thesewereareaswheretransmissiontrippingandotheractionsleft
themwithgenerationandloadinapproximatebalance.
Bythefirstevening(August14),powerhadbeenrestoredto:
Albany,NewYork,anditssurroundings;
NewLondonCounty,Connecticut;
ParrySound,Ontario
ManyareasoftheNiagaraRegioninOntario;
AreasofOntarionearNiagaraFalls(suppliedfromthecityofNiagaraFalls,
Ontario,whichneverlostpower);
PartsofSouthwesternOntario,particularlyareasneartheBruceNuclear
PowerPlant,lostpowerforonly48hours;
PartsofdowntownToronto,Mississauga,London,Ontario,Cornwalland
Pembroke;
PortionsofwesternOttawaincludingKanataandsouthtoKingston;
ThreequartersofthemillionsofcustomerswhohadlostpowerinNew
Jersey;and
PartsofPennsylvania,Ohio,Michigan,andLongIsland.
ThatnightsomeareasofManhattanregainedpoweratapproximately05:00(August15),the
NewYorkCityboroughofStatenIslandregainedpoweraround03:00onAugust15,and
theNiagaraFallsareaat08:00.
ByearlyeveningofAugust15,twoairports,ClevelandHopkinsInternationalAirportand
TorontoPearsonInternationalAirport,werebackinservice.
Page 16 of 69
DGAConsulting
HalfoftheaffectedpartofOntariohadpowerbythemorningofAugust15,thoughevenin
areaswhereithadcomebackonline,someserviceswerestilldisruptedorrunningatlower
levels.Thelastareastoregainpowerwereusuallysufferingfromtroubleatlocalelectrical
substationsnotdirectlyrelatedtotheblackoutitself.
ByAugust16,powerwasfullyrestoredinNewYorkandToronto.Torontossubwayand
streetcars(trams)remainedoutofserviceuntilAugust18topreventthepossibilityof
equipmentbeingstuckinawkwardlocationsifthepowerwasinterruptedagain.Powerhad
beenmostlyrestoredinOttawa,thoughauthoritieswarnedofpossibleadditional
disruptionsandadvisedconservationwhilerestoringpowercontinuedforotherareas.
Ontarianswereaskedtoreducetheirelectricityuseby50%untilallgeneratingstations
couldbebroughtbackonline.Fourgeneratingstationsremainedoutofserviceonthe19th.
Illuminatedbillboardswerelargelydormantfortheweekfollowingtheblackout,andmany
storeshadonlyaportionoftheirlightson.
New York
TheNYISORestorationPlanreliesonblackstartfacilitiesatthreelocationsinthestateto
energizethebasicminimumpowersystem.9Twooftheselocations,theNiagaraandSt.
Lawrencehydrogenerationfacilities,remainedinservicefollowingtheevent.
At16:27,theNYISOinstructedNewYorkutilitiestobeginblackstartprocedures.Oneof
them(NYPA)beganblackstartproceduresatakeypumpedstoragegeneratorbystripping
thenorthandsouth345kVbuses.Theseprocedureshadbeenthoroughlyreviewedin
simulateddrillsasrecentlyasJune.Implementingthemprovidedamorecertainsetof
initialconditionsfromwhichtobeginrestorationofcritical345kVfacilities,whichhadbeen
lostalongtheHudsonValley.Theonlycomplicationencounteredduringthissequencewas
aninabilitytosynchthetwoblackstartunitsontoa345kVline.Thislinecouldnotbe
closedattheplantduetothelargevoltagedisparitybetweentheplantbusandthe345kV
bus.Systemvoltageswerefurtherstabilizedwhenanother345kVlinewasrestoredat
19:05.Subsequently,alltheconnected345kVlineswererestoredinthenext15minutes.
OneoftheNYISOsfirstobjectiveswastoresynchronizetheNewYorktransmissionsystem
withthePJM500kVinterconnectioninPennsylvania,torestorenormalfrequencycontrolto
9. ThisNewYorksectionisbasedonmaterialintheNYISOsFinalReport:OntheAugust14,2003Blackout,
February2005.
Page 17 of 69
DGAConsulting
theWesternNewYorkIsland.Initialsynchronizationoccurredat18:52whentheNYISOwas
abletocoordinatethebalanceofgenerationandloadlevelsattherequiredfrequencyforthe
synccheckrelaytooperateandparallelthetwosystems.
TheNYISOalsodirectedConEdison(NewYorkCity)personneltomanuallycloseintothe
PJM500kVgridviasynchroscopeoperationat19:06.UltimatelyasecondNewJerseytie
wasrestoredat19:08providingamoresecureinterconnectionwiththePJM500kVand
345kVtransmissionsystems.Followingtheseevents,thefrequencycontrolintheWestern
NewYorkIslandreturnedtonearnormal.
InpreparingtosynchronizewithISONE,voltageswerestabilizedintheeasternareaofNew
York.AftersuccessfullyreclosingwithPJM,systemfrequencyinNewYorkstabilized.The
effortwasthentostrengthentheNewYorksystemtoprovidemorestablevoltagestotie
intoISONE.ThiswasaccomplishedwhilerestoringlinesintotheConEdisonarea.
TheNYISOandISONEoperatorscoordinatedtherequiredactionswiththeirassociatedTOs.
ISONEusedapumpedstoragehydrofacilityneartheNewYorkbordertosynchronize
usingasynchroscope.Theconnectionwasmadeat01:53onAugust15th.
Throughoutthisevent,loadandgenerationbalancewasessential.TheNYISOoperators
instructedallTOstonotifytheNYISOofallloadrestorationsandgeneratoravailability.The
TOswereinstructedtomatchloadwithgenerationasitbecameavailable.Voltagecontrol
wasalsoanimportantconsideration.Insomecases,loadwasrestoredfromgeneration,in
otherpartsofNewYorkloadwasrestoredtocontrolhighvoltagesduringlinerestoration.
Thisprocessofcoordinationwasverysuccessfulduetotherepeatedtrainingforthistypeof
eventandtheexcellentcommunicationsbetweentheNYISOandtheTransmissionOwner
operators.
By06:00onAugust15th,56%oftheloadhadbeenrestoredinNewYork.At7:35theNYISO
activatedvariousenergyandloadmanagementprogramsandalsorequestedthepublicto
voluntarilycurtailelectricuseandannouncedtemporarywaiversofairemissions
limitations.TheNYISOwaspreparingforthemorningloadtobeginpickingup.At08:00,in
aconferencecall,theNYISOnotifiedTOsthatloadsheddingmightberequiredduetothe
morningloadpickup.Thegroupagreedthattheexistingloadshedallocationprocess
wouldbemodifiedandthattheloadshedallocationswouldbecalculatedbasedonthe
percentageofthecurrentTOloadtothetotalNewYorkloadatthattime.
Page 18 of 69
DGAConsulting
Michigan
InMichiganthereweretwomajorutilitiesaffectedbytheblackoutConsumersEnergyand
DetroitEdison.10DetroitEdisonservesDetroitandthesurroundingareainsoutheastern
Michigan.ConsumersEnergyservesareastothewestofDetroitEdison.(DetroitEdison
washitmuchharderbytheblackoutthanConsumersEnergy.)Thetransmissionsystemis
ownedandoperatedbytwoindependentcompaniestheMichiganElectricTransmission
Company(METC)forthetransmissionsystemformerlyownedbyConsumersEnergyand
theInternationalTransmissionCompany(ITC)forthetransmissionsystemformerlyowned
byDetroitEdison.
MISOwasalsoinvolvedtheregionaltransmissionorganizationcoveringallorpartsof
Indiana,Illinois,Iowa,Kansas,Kentucky,Manitoba,Michigan,Minnesota,Montana,North
Dakota,Ohio,Pennsylvania,SouthDakota,andWisconsin.(SeeFigure2onpage10,
above.)
Consumers Energy
Restorationeffortsbeganimmediatelyfollowingtheevent.Localheadquartersinthe
affectedareaswereinstructedtoremainopen.Aconferencecallwasestablishedat17:15to
determineinitialactions.Subsequentcallswereheldeverytwotothreehoursthereafter.
IndependentcallswerealsoheldwithMETConasimilarschedule.
OntheConsumersEnergy/METCsystemthereweresignificantgeneratoroutages,numerous
lineoutages.Thereweretwomajorareaswithoutpower,theLansingareaandthe
southeastcornerofConsumersEnergysserviceterritory.
Immediatelyfollowingtheevent,ConsumersEnergystartedgenerationinresponsetothe
lossofunits.ConsumersEnergybelievedatthattimeitwasundergenerating,but
interconnectionfrequencycontinuedtobeabove60Hertz,whichwouldgenerallybean
indicationofovergeneration.Inconsultationwithtransmissionoperators,Consumers
Energymaintaineditsgenerationleveluntilthestatusofthesystem,bothinMichiganand
inneighboringareas,couldbeassessed.Between17:00and19:00poweroutputfromthe
LudingtonPumpedStoragefacilitywasreducedinordertomoderatehighfrequencylevels
andmanageavailablestoredwaterforlaterrestorationneedsofDetroitEdison.Consumers
10. ThisMichigansectionisbasedonMichiganPublicServiceCommissionsReportonAugust14thBlackout,
November2003.
Page 19 of 69
DGAConsulting
Energyalsoobtainedadditionalsuppliesofelectricityfrominstateindependentpower
producersandthemajorutilitytoitssouth,AmericanElectricPower(AEP).
Restorationeffortsfollowedblackstartprocedures;beginningbyassessingthe138kVand
46kVbreakersthatwereopen.Theopenbreakerswereplottedonageographicmapofthe
electricsysteminordertodeterminetheboundariesoftheaffectedareas.SystemControl
Centersthenbegantheprocessofopeningupallbreakerscontainedwithintheaffectedarea
viasupervisorycontrolanddataacquisition(SCADA)andfieldpersonnel.
ThereturnofgenerationattheWhitingfacility(a328MWcoalplantbuiltin1952thatalso
hasa15MWsimplecyclecombustionturbine)andrestartinggeneratorsatKinderMorgan
powerplant(a540MWgasfueledcombinedcycleplantbuiltin2002)wereatoppriority.
Theseunitsprovidebothlocalpowersupplyandareavoltagesupport.Nearlyallthe
138kVsystemwasrestoredby19:25.Duringrestorationofthe138kVsystemsome
46kVand138kVconnectedloadwasalsorestored.
Asgeneration,particularlytheKinderMorganpowerplant,beganrampingtowardfull
output,the46kVsystemwasrestoredintheaffectedarea.By22:05all46kVlineshad
beenenergizedandallloadwasreturnedtoservice.
AccordingtoConsumersEnergysOutageManagementSystem,upto118,400customers
wereoutofserviceduringthe16:00through22:00onAugust14th.
At22:30animportant138kVLinetrippedanddidnotrecloseduetolossofsubstation
power.Thisresultedinlargeflowsontheremainingtwocritical138kVlines,causingthem
toopenattheirsourceends.Thesystemwithinthesubjectgeographicareawasthenin
nearlythesamestateasitwasfollowingtheprimary16:09outage.
The138kVsystemwasrestoredagainby00:55andthe46kVsystemalongwithallofthe
connectedcustomerswasrestoredby01:35.
ConsumersEnergypersonnelhandledseveralreliabilityconcernsoverthenexttwodays.
Theseincludedproblemsinadheringtothederatedcapabilityofthecritical345kVline,the
clearancestatusof138kVlineslocatedwithintheaffectedarea,andlargepowerflows
betweentheMETCandITCsystems.Inaddition,continuedhotweather,unitoutagescaused
bytheevent,anduncertainpoweravailabilitytosupplementConsumersEnergysown
internalgenerationledtoaforecastofadeficiencyinConsumersEnergysoperating
reserve.
Page 20 of 69
DGAConsulting
ThefinalconcernwaspossibleseparationbetweentheConsumersEnergy/METCandthe
DetroitEdison/ITCsystemsduetoanyoneofthreesinglecontingenciesinvolvingtielines
betweenthetwosystemson16August.WithDetroitEdisonbeinggenerationdeficient,it
wasdependentupontheConsumersEnergy/METCsysteminterfaceforpowersupply.With
powerflowsbetweenthetwosystemsreachingthe3,000MWrange,analysisindicateda
singlecontingencywouldloadothertiesaboveemergencycapabilities.Thiscouldstarta
cascadingoutageresultinginseparation.Anumberofoptionswereimplementedto
preventthisfromoccurringincludingpatrolsofthetielinesidentifiedbytheanalysis.
Detroit Edison
TheDetroitEdisonserviceterritorywideoutageinvokedtheutilitysblackstartprocedures.
Theseprocedureswereinitiallydevelopedafterthe1965outage.Theydirectedallthe
availablefieldoperationsstafftotheproperlocationstosupporttherestorationeffort.
(ConsumersEnergyalsohadsimilarproceduresfromthattime.)Giventhe
telecommunicationandtrafficissuesthatoccurredimmediatelyaftertheincident,these
proceduressavedvaluabletimerestoringthesystem.
Theconventionalmobileandlandlinephonesystemsarenotdesignedforemergencies.The
volumeofcallsoverloadsthesystemsanditisoftenimpossibletoevengetadialtoneto
makeacall.Similarly,aftersuchanevent,oncepeoplerealizethesituationtheytaketo
theirautomobilesallatonceandtrafficquicklycomestoastop.Havingpersonnelknow
wheretopositionthemselvesbeforetrafficsnarls,allowsthemtobeinplacewhenneeded
aspartoftherestorationprocess.ThisisnotasituationuniquetoDetroit;itcommoninany
urbanareawhenablackoutoccurs.
DetroitEdisonfacedthedifficultprospectofrestartingitsentiregenerationfleet.Basically,
theylostalltheirgenerationabout9,500MW.Thisincludedabout7,500MWincoalplants
builtbetween1949and1985,a780MWgasfueledplantfrom1979,anda1,130MWnuclear
unit(1988).Oneofthecoalunitstrippedtohouseloadandwasrestoredwithinafew
hours.A330MW(fourgasturbines)independentpowerproducerwasreturnedtoservice
at20:15.Alargegasunitandanothercoalunitwererestoredbymiddaythenextday.
DetroitEdisonpersonnelalsoresolvedissuesatvariouspeakingfacilitiestoprovide950MW
ofadditionalcapacityonAugust15.11
11. AdditionaldetailscanbefoundinChart3.8oftheMichiganPUCblackoutreport.
Page 21 of 69
DGAConsulting
TheotherDetroitEdisongenerationtooklonger.
Alltheunitswereinspectedfollowingtheoutageandfourunitshadruptureddiscsthat
werereplaced.Thesmallercoalunitswererestoredbeginningasearlyas03:30onAugust
15th.Bytheafternoonofthe15th,sevenoftheseunits,totaling1,420MW,werebackin
service.TheremainingunitswererestoredbytheafternoonofAugust19thfourdaysafter
theblackout.
InadditiontoDetroitEdisonsgeneration,purchasesfromoutsidetheserviceterritoryin
coordinationwithindependentpowerproducerswerecrucialtotimelyrestoration.
Lessons learned
TheNorthAmericanElectricReliabilityCorporation(NERC)SteeringGrouprecommended
threecategoriestoaddresstheshortcomingsidentifiedintheinvestigation:
1. Actionstoremedyspecificdeficiencies:specificactionsdirectedtoFirst
Energy,MISO,andPJMtocorrectthedeficienciesthatledtotheblackout.
2. StrategicinitiativesbyNERCandtheregionalreliabilitycouncilstostrengthen
compliancewithexistingstandardsandtoformallytrackcompletionof
recommendedactionsfromAugust14,andothersignificantpowersystem
events:
a. StrengthentheNERCComplianceEnforcementProgram;
b. Initiatecontrolareaandreliabilitycoordinatorreliabilityreadinessaudits;
c. Evaluatevegetationmanagementproceduresandresults;and
d. Establishaprogramtotrackimplementationofrecommendations.
3. Technicalinitiativestopreventormitigatetheimpactsoffuturecascading
blackouts:
a. Improveoperatorandreliabilitycoordinatortraining;
b. Evaluatereactivepowerandvoltagecontrolpractices;
c. ImprovesystemprotectiontosloworlimitthespreadoffutureCascading
Outages;
d. Clarifyreliabilitycoordinatorandcontrolareafunctions,responsibilities,
capabilitiesandauthorities;
e. Establishguidelinesforrealtimeoperatingtools;
Page 22 of 69
f.
DGAConsulting
Evaluatelessonslearnedduringsystemrestoration;
g. Installadditionaltimesynchronizedrecordingdevicesasneeded;
h. Reevaluatesystemdesign,planningandoperatingcriteria;and
i.
Improvesystemmodelingdataanddataexchangepractices.
2013 Sarawak
On27June2013theMalaysianStateofSarawaksufferedatotalgridfailureresultingina
statewideblackout.12At17:36thesystemfrequencystartedtodecay,fallingbelow47Hz
within10seconds,followedbycascadingtrippingofgenerationandtransmissionwithinthe
Sarawakpowergrid.

Itwasafairlytypicalsummerafternoon.Beforetheblackoutsystemloadwasabout
1,555MWwith2,034MWofgenerationoperatinginthesystem.Thisisalittlelessthan90%
ofthemostrecenttwelvemonthpeakload.Systemfrequencywas50Hz(normal).Mostof
thegeneration(57%)wasatarelativelynewhydroelectricplantBakun.
TheloadandgenerationbalanceofthefiveregionsinSarawakjustbeforetheblackoutare
showninFigure7.Thefigureshowsthatfourareasareimportingpower,withBakun
exporting.Allfiveareashadsomelocalgenerationoperating.Theseloadandgeneration
balanceswereimportantregardingloadsheddingandformingelectricalislands.
12. Muchofthissectionisbasedon27June2013SarawakBlackoutDNVKEMAIndependentReview,Ministryof
PublicUtilitiesSarawak,Finalreport,PreparedbyKEMA,Inc.,6October2013.
Page 23 of 69
DGAConsulting
Figure 7: Regional load and generation just before blackout
The blackout
At17:36GenerationattheBakun,whichwasdelivering885MWtothesystem,droppedits
outputby662MWinabout10secondscausingsystemfrequencytodropbelow47.5Hz.To
protectalltheothergeneratorsinthesystemfromdamage,safetytrippingwas
automaticallyactivated,resultingintheshutdownofmostotherpowerstationsinthegrid.
Theunderfrequencyloadsheddingsystemactivated,sheddingabout500MWofloadinfive
steps.Frequencycontinuedtodecline,however,andthesystemwentblack14secondsafter
theinitialeventbegan.
TheinitiatingeventwasthesuddenrampdownofgenerationatBakundam.Itappeared
thatsomeoneinsidetheplantaccessedthepowerplantcontrolsystemandinadvertently
initiatedtheshutdown.Normally,thecontrolsareconnectedonlytoaspeciallocal
computercommunicationnetwork.Onthedayoftheblackout,thethreeunitsthatramped
downwerealsoconnectedtothenormalinternalofficecomputernetwork.Itappearsthat
someoneattheplantinadvertentlyinstructedthethreeunitstorampdown.Thereisno
evidencethatthiswasanykindofcyberattack.
Page 24 of 69
DGAConsulting
Restoration
RestoringtheSarawaksystemloadtookmorethansixhoursfollowingtheblackout.Nearly
halfofthistimealmostthreehourswasduetooperatorerrorsandequipment
malfunctionsatBakunduringrestoration.
TherewerealsosomedifficultiesattheBintulupowerplantthatdelayedrestoringlocal
loadsintheBintuluarea.Whilethreeunitswereoperatingimmediatelyaftertheblackout,
theotherBintuluunitscouldnotberestartedwithoutexternalpower.
Sarawakrestorationplansaredesignedtoguideindividualgeneratingplantoperatorsand
thesystemoperatorinquicklyrestoringthesystem.Eachofthesehasaplanwithspecific
stepsforrestoration.
Briefly,Sarawaksoverallblackstartrestorationplanincludes:
1. Determinethepostblackoutsystemandequipmentstatus;
2. Communicatewithrelevantparties;
3. Mobilizepersonnel;
4. Preparegeneratingplantsandthegridsystemforsystematicrestoration;
5. Reenergizeelectricislands;
6. Resynchronizetheelectricislandstorestoretheinterconnectedgridsystem;
and
7. Confirmthatthesystemisbacktonormalandallloadsareenergized.
Oncethegeneratingunitsarestartedatalowlevel,theyarereadyforstep5.Closing
distributionfeedercircuitbreakersconnectscustomerload.Inthiswayeachgenerator
increasesitoutputbyservingloadinitsgeographicarea.Thesearetheelectricislandsas
thelocalgenerationandloadisnotconnectedtogeneratorsinotherareas.
InSarawak,theplanistoformthreeelectricislands:
1. Kuching/SibuSarikel;
2. Bintululocal;and
3. BakuntoMiri275kV.
Aftertheelectricalislandsarerunningatareasonablelevel,theyarereadytostart
interconnectingwiththetransmissionnetworkandtheotherislands.Thisisstep6,above.
Page 25 of 69
DGAConsulting
Duringthe27Juneblackoutevent,theSibuSarikelelectricislandformed(almostexactlyas
planned),butnottheBintuluorBakun/Mirielectricislands.Kuchinghasagas/dieselpower
plantandBatangAIisahydroelectricplant.Thesewerebothconnectedtothesystem,
restoringloadintheKuchingareawithinabout30minutesoftheblackout.
Followingtheblackout,threegasfuelledgasturbineunitsatBintuluremainedoperatingat
fullspeedwithnoload.Theotherfiveunitstrippedoff.About45minutesaftertheblackout
(at18:25),the33kVbuswasenergizedandauxiliarypowerrestoredforfouroftheseunits.
Duringthenextthreehours,powerwaspartiallyrestoredtoloadintheBintuluarea.About
threehourslater(21:35,4hoursaftertheblackoutstarted)powerwasrestoredtoBakunthat
allowedauxiliarypowertoberestoredtotheremainingBintuluunits.
DuringtheovernighthourstheremainingBintuluunitswererestartedandsynchronized
withthegrid.Thisallowedloadtobefullyrestoredintheearlyhoursofthenextday.
Asmentionedabove,problemsatBakundelayedrestorationbyaboutthreehours.While
therewereanumberofblackstarterrorswithintheplant,themostsignificantwere
operationerrorsinthe275kVswitchyardattheplant.TheBakundamandswitchyardare
remoteathreehourdrivefromBintulu.Thismeantthatpowerplantpersonneloperated
breakersandswitchesattheBakun275kVswitchyard.Sincetheplantisremotewithlong
transmissionlines(about100km),voltagecontrolisanessentialelementinreconnectingthe
planttothemainSarawakgrid.SwitchingerrorsatBakuncausedsignificanthighvoltages
thatsetbackthenormalrestorationprocess.
Lessons learned
Maintainingcontinuityofplantpersonnelatremotepowerplantscanbeacontinuing
problem.TheBakundamisquiteremote,withnoeducation,entertainmentorsimilar
optionsforemployeesortheirfamilies.Nodoubtsomeoftheproblemsthereresultedfrom
thechurnofstaffattheplant.
Personnelmustbekeptcurrentintheirblackstarttraining.
2008 Oahu, Hawaii

About18:30on26December2008alightningstormknockedoutpowerthroughoutthe
HawaiianislandofOahuaftersunset,forcingPresidentelectBarackObamaandhisfamily
tospendthenightattheirseasidevacationhomeinthedarkaswindssweptinfromthe
ocean.Utilitycrewsrestoredpowertothepropertyaround06:00,andbymiddayelectrical
Page 26 of 69
DGAConsulting
servicewasbackformanyoftheislands900,000residentsandthousandsoftourists,whose
highrisehotelsstoodmostlydark.

Justpriortothelightningstorm,HawaiianElectricCompanys(HECOs)138kVtransmission
and46kVsubtransmissionsystemswereintheirnormalconfigurationwithalllinesin
service.Generationoperationwasnormal(1,300MW)toservethe1,040MWload.Spinning
reserveswere260MW,80MWabovetheHECO180MWminimumspinningreserve.HECOs
dispatchcenterandpowerplantswereproperlystaffed.LocationsofOahugenerationare
shownonFigure8andtheirstatusjustbeforetheblackoutisshowninTable2.
Figure 8: Oahu generation
Page 27 of 69
DGAConsulting
Table 2: Summary of HECO generation

Unit
Rated (MW)
Available (MW)
Actual (MW)
2 HECO CTs
103
103
2 IPP CTs
164
164
162
18 HECO diesels
30
30
10 HECO steam
863
863
607
3 IPP steam
270
270
268
The blackout
TheIslandofOahuexperiencedaseverelightningstormonDecember26,2008thatlasted
fromapproximately18:00to18:50.DuringthisstormtheHECOtransmissionsystem
experiencedfiveseparateshortcircuiteventscausedbylightningstrikesonornear138kV
transmissionlines.
Oneofthesewasa3phasefaultonanimportant138kVtransmissionlinejustoutside
Honolulu.ThisfaultmomentarilydepressedvoltagesacrosstheHECOsystemtothepoint
whereapproximately145MWofvoltagesensitivecustomerequipmentautomaticallyshut
down.Thisvoltagedipalsoimpactedthenormaloperationofcertaingenerationplant
auxiliarysystemsleadingtopowerplanttripping.This,inturn,initiatedasequenceof
powersystemeventsduringthenextfewminutesthatultimatelyleadtotheislandwide
blackout.
Assystemfrequencydeclined,Underfrequencyloadshedding(UFLS)operatedasexpected
infivestages.Distributeddieselgeneratorsbegantransferringfromautotomanualcontrol
andshuttingdown,mostlikelyduetothefrequencydrop.
Restoration
PowerEngineers(aUSengineeringconsultingfirm)foundthatHECOsrestorationefforts
wereprudentandrestoredservicetoitscustomersexpeditiouslyconsideringthe
circumstances.Figure9showsthetimelineofgeneratorstartsasservicewasrestored
betweenabout19:00and15:00thenextdaywhenallcircuitswererestoredalongwith96%
ofcustomers.Afewpocketsremainedwithoutserviceuntilthenextmorning.
Page 28 of 69
DGAConsulting
Figure 9: Oahu restoration timeline
Therestorationtimelineshowsseveralfailedgenerationrestartsuntiljustafter01:00.
Generationandloadarethenrestoredatafairlylinearratethrough10:00whenmorethan
80%oftheloadwasrestored.
ThePowerEngineersreviewfoundthatHECOcouldnothavereasonablyanticipatedor
preventedthedamagingeffectsandinstabilitycausedbythelightninginitiated3phase
shortcircuittopreventthepoweroutagefrominitiallyoccurringorfromitbecoming
islandwideunderthecircumstances.13ThereportcontinuestheHECOsystemwasin
properoperatingconditionandwasappropriatelystaffedbypersonnelatthetimeofthe
lightningstorm.Thesystemoperatedappropriatelyunderthecircumstances.Inresponseto
thevariouslightningstrikes,automaticprotectiverelaysthataredesignedtosensea
disturbanceonHECOstransmissionsystemoperatedasdesigned.
Lessons learned
Thisblackoutrepeatstheexperiencewithotherblackoutswhenoperatorsarechallengedin
restoringserviceduringthefirstfewhoursfollowingtheblackout.
13. PowerEngineers,OutageReport,31March2009,pages34.
Page 29 of 69
DGAConsulting
2003 Italy
Earlyinthemorningof28September2003,Italyexperiencedamajorblackoutaffectingall
ofItalyexcepttheislandsofSardiniaandElba.PowerwasoutinItalyfor12hoursand
partofGeneva,Switzerlandfor3hours.Itwasthelargestblackoutintheseriesofblackouts
in2003,affectingatotalof56millionpeople.ItwasalsothemostseriousblackoutinItalyin
70years.
Thenightof27September2003isthenightoftheannualovernightNuitBlancheinRome.
Thus,manypeoplewereonthestreetsandallpublictransportationwasstilloperatingat
thetimeoftheblackoutdespitebeingverylateatnight.Theblackoutcausedthecarnivalto
endearly.Severalhundredpeopleweretrappedinundergroundtrains.Coupledwith
heavyrainatthetime,manypeoplespentthenightsleepingintrainstationsandonstreets
inRome.
ThroughoutItaly,110trainswerecanceled,stranding30,000people.AllflightsinItalywere
alsocancelled.Policedescribedthesceneaschaosbuttherewerenoseriousaccidents.
Theblackout,however,didnotspreadfurthertoneighboringcountries,suchasAustria,
SloveniaandCroatia,whichareconnectedtoItaly.

Intheyearsbefore2003,ItalyselectricityimportsgrewsharplyduetoItalyssignificantly
higherelectricproductioncoststhantherestofEurope.ThefactthatSwitzerlandwasnot
integratedwiththeEuropeanelectricitymarketandoperationmeantthattheincreasing
importsintoItalywerealsoflowingonunscheduledparallelpathsthroughSwitzerland.
Thismeantthatloadsoncrossbordertransmissionlinesoftendeviatedfromscheduled
exchangeswithevergrowingamountsflowingontheSwisstransmissionlines.The
resultingpowerflowswerenotalwayswellcoordinatedbetweentheEuropeansystemand
Switzerland.
Atthetimeoftheevent,theItalianloadwasverylow27,444MW,including3,487MWof
pumpload.Italywasgenerating20,493MWandimporting6,951MWasshowninTable3.
Thetablealsoshowsthesignificantdifferencebetweenscheduledandactualflowsonthe
interconnectionswithSwitzerlandandFranceand,toalesserextent,withSlovenia.
Page 30 of 69
DGAConsulting
Table 3: Italian imports at 03:00 28 September 2003

Imports
Interconnection
Scheduled
Actual
Excess
Switzerland
3,068
3,610
542
France
2,650
2,212
-438
Slovenia
467
638
171
Austria
223
191
-32
Greece
285
300
15
6,693
6,951
258
Total
The blackout
Earlyinthemorningof28September2003Italywasimportingabout6,700MW(25%of
demand).14TheItalianpowersystemwasoperatingundern1securityconditions,andwas
capableofcorrectlydealingwiththelossofanyindividualelementofthegrid,including
anyinterconnection,orthelossofthelargestunitinservice.
At03:01,a380kVinterconnectionbetweenSwitzerlandandItalytripped.Thelinetripped
duetoaflashoverwithtrees.Severalattemptsmadebytheautomaticreclosingfacilities
wereunsuccessful.Amanualreclosingattemptmadeat03:08alsofailedbecauseofan
overlyhighphaseangle(42)betweenItalyandSwitzerland.
Anearbyparallelcircuitthenoverloaded.Thisoverloadwasacceptableinsuchemergency
circumstances,butfornomorethan15minutesaccordingtooperatingstandards.
Unfortunately,operatoractionsdidnotreducetheoverloadintime.After24minutes
(03:25),thisoverloadedlinealsotrippedwhenitflashedoverwithatree.
Aparallel220kVlinethentrippedimmediately.AlmostsimultaneouslytheItaliangridlost
itssynchronismwiththeUnionfortheCoordinationofElectricityTransmission(UCTE)main
14. MuchofthisItaliansectionisbasedontheUCTEreport:FinalReportoftheInvestigationCommitteeonthe28
September2003BlackoutinItaly,April2004.
Page 31 of 69
DGAConsulting
grid.15AllremaininginterconnectionsbetweenItalyandUCTEweredisconnectedbythe
normalactionofprotectivedevices.By03:26Italywasanelectricalisland.
TheseparationsequenceoftheinterconnectionsisshowninFigure10.Whilethefirsttwo
linesseparatedwithin24minutes,theother7interconnectionsopenedwithinabout90
seconds.
Figure 10: Separation sequence of Italian interconnections
Atthispoint,theItaliansystemhadgenerationequaltoonly75%ofitsload.Evenso,
generationwasnearlyabletomeetload,however,itwasnotenoughtoarrestthefrequency
decline.Inthethreeminutesafterseparation,thefrequencydeclinedto47Hzwhenanearly
completecollapseoftheItaliannetworkwasinevitable.
Someunderfrequencyrelaysactivatedtoformelectricalislandsoftwowelldefinedportions
ofthenetworkandtherelatedloadinsouthernItaly.Athermalunitshouldhavefedeach
island,butonlyonegeneratingunitwasinanormalconfiguration,so,onlyonesuccessfully
islanded.
15. UCTE,wasanassociationofTSOsin23countriesacrosscontinentalEurope.UCTEisresponsiblefor
coordinatingtheoperationanddevelopmentofelectricitytransmissiongridinandamongitsmember
countries
Page 32 of 69
DGAConsulting
Thirtyonethermalunitsinitiatedthesequencetotriptohouseload.Onlyeightofthem
successfullycompletedthesequenceandremainedinisolatedoperationonhouseload.
Restoration
TherestorationplanforItalyisasetofcodedguidelinesthatareusedbyoperating
personneltorestoresupplyafteralargeareaincidentorblackout.Thisplanisbasedon
severalrestorationpathsdesignedtoworkinparallel:torestoretheauxiliaryservicesof
shutdownplants,toreconnectthethermalpowerplantsthatsuccessfullyislandedor
trippedtothehouseload,andtostabilizetheloadofsuchplants.
Therestorationpathsdesignedandusedforrestoringthenetworkaftertheblackout
were:
13innorthernItaly
9incentralsouthItaly
4inSicily
Thereare24hydroorgasturbineunitswithblackstartcapabilityusedtorestorethepaths.
Whilenospecificperformanceindexexists,theoverallsuccessoftheplandependsonfour
keyvariables:
1. Thenumberofavailablethermalunits,operatingontheirhouseloadafter
separationfromthegrid;
2. Thereadinessofhydroandgasturbineunitstoperformblackstart;
3. Thereliabilityoftelecontrolandtheirtelecommunicationsystemstooperate;
and
4. Theavailabilityofhydro,conventionalthermalandgasturbineunits.
Inthisblackout,mostoftherestorationprocesseswereperformed
satisfactorilyincomparisontotheseverityofthetotaloutage.Even
Restored
load (%)
so,ittookmorethan18hourstofullyrestoreserviceascanbeseen
50
6:30
inthetableontheright.
70
10:00
99
15:00
100
18:12
Anoverallviewofthetimingoftherestoredloadandthesupply
Time
Hr:min
sourcescanbeseeninFigure11.
Page 33 of 69
DGAConsulting
Figure 11: Italy restoration timeline
ThepostblackoutreviewbytheUCTEfoundthattheblackout:
Wasnotcausedbysomeextraordinaryoutofcriteriaeventsuchasasevere
storm,acyberattack,orsimultaneouslightningstrikesonseverallines,etc
WastriggeredbycausesinSwitzerlandwheretheinitialeventswereoutof
reachforactionbytheItalianoperators.
Therewerecountermeasuresforreturningthesystemtoasecurestateafter
thefirstcontingency(fromapurelytechnicalpointofview),buthuman,
technicalandorganizationalfactorspreventedthesystemfromreturningtoa
securestate.
Italiansystemrestorationwasperformedsuccessfully,however,itwould
havebeenshorterifmoreunitshadsuccessfullyswitchedtohouseload
operationorhaveperformedblackstartcorrectly.
Therestorationoccurredin4stages:
1. Stage1(03:28to08:00)diagnosisandNorthernArearesupply;
2. Stage2(08:00to12:00)Intermediatesteps
3. Stage3(12:00to17:00)ThecompleteresupplyofMainland
Page 34 of 69
DGAConsulting
4. Stage4(17:00to21:40)ThefinalstageandtheresupplyofSicily
Italian restoration stage 103:28 to 08:00
Approximately3hoursaftertheblackout(06:30):
NorthwesternItalywasalmostcompletelyenergizedandreconnectedtothe
Frenchgrid;
EasternMilanareawassynchronizedwithSwitzerland;
AllnorthwesternItalythermalplantbuseswereenergizedand750MWwere
synchronizedandoperatingatminimumoutput;
EasternVeniceareawasfedbySlovenia;and
PartofthenorthernFlorenceareahadbeenconnectedtoLombardy.
Around08:00attheendofthefirststage:
ThenortheasterngridwasnotyetmeshedwiththerestoftheNorthern
SystembutitwassynchronouswithitviatheUCTEsystem.Thesubstations
inthatareawereenergizedandstableenoughtostartgeneratorauxiliary
services.
Inthecentralsouthernarea,nosignificantprogresshadbeenmade.
Duringthefirststagesomefailuresoccurred,suchasdifficultiesinstartingblackstartunits,
voiceanddatacommunicationproblemsandlackofinformationfromthefield.
TheSCADAsystemoftheNationalControlCenterinRomelostcompletevisibilityofthe
datarecordedintheFlorenceareaandnorthernRomeareafrom06:31to13:17.Two
strategieswereadoptedinthecentral,southernareasandinSicily:first,createasmany
islandsaspossibleinthesouthernareas,and,secondproceedstepbystepfromNorthto
SouthrelyingoninterconnectionwithUCTEandonoperatinghydroelectricresources.
Figure12showsthesituationattheendofStage1.
Page 35 of 69
DGAConsulting
Figure 12: Italy at the end of Stage 1 restoration

Restoringthepowersystemcontinuedataslowerpacethandesiredbecauseofswitching
difficulties,mainlyduetotelecontrolsandtominorproblemsofdisconnectorsinsomekey
substations.Additionally,astheaccidenthappenedbeforethepumpedhydroplantshad
completedfillingtheirupstreambasins,availablehydroenergydecreased.Tocompensate
importswereincreased.
ThemarketoperatoraskedDistributorstointerruptindustrialloadsandimplementthe1st
levelofloadshedding(rotatingloadsheddingplan)from11:00to18:00inthenorthernand
centralnorthernregionsthatwereelectricallyrestored,buttheresponsewasnoteffective.
Page 36 of 69
DGAConsulting
IntheRomearea,at09:28afterafewattemptstodelivervoltagefromahydroblackstart
unitwhichhadfailedowingtovoltageinstability,another310MVAuniteastofRomewas
startedinordertobettercontrolthesamerestorationpath.From10:06theresupplyofthe
metropolitanareaofRomebegan.
Attheendofstage2,loadintheNorthernareawaspracticallyrestored.Althoughthe
380kVgridwasreenergizedtoRomeandtheAdriaticbackbonewasenergizedupto
Marcheregion,theresuppliedloadwasstilllow.
IntheSouthernareatwoelectricislandsinsouthernRomeandintheextremesouthwest
regionofCalabriawereinterconnectedandexpanded.
Figure13showsthesituationattheendofStage2.
Page 37 of 69
DGAConsulting

Onlyonethermalpowerplantinthesouthwasstillnotreenergized.Thestatusofthegrid
andthelackofgenerationinCentralSouthernItalyinducedhighpowerflowsfromNorth
toSouth,creatinganatrisksituation.Asituationthatbecameworseduringtheevening
peakperiod.ImportedenergyfromGreecebecamecriticalbecauseitenabledthepower
flowsbetweennorthernsouthernmacroareas,toberelievedofrestoringsecurity
conditions.ThesituationwasresolvedbyrestoringthelinkwithGreece;allowingimports
ofupto500MW.
Page 38 of 69
DGAConsulting
BytheendofStage3alloftheItalianpeninsulaandpartsofSicilywerereenergizedas
showninFigure14.

AllthevariousproblemsdescribedabovewereamplifiedintheSiciliangrid.
Afterseveralunsuccessfulattemptstorestoretheserviceindependently,withoutthe
mainlandinterconnection,itwasdecidedtosupplytheislandfromCalabria.Thiswasdone
at16:38buttheexchangecouldnotexceed200MW.Thus,theresupplytimewaslimitedby
Page 39 of 69
DGAConsulting
theneedtoadjusttheloadrampinItaly.WiththeSiciliangridrestored,theItalianpower
systemwasagainundercontrolandemergencyconditionsendedat21:40,alittlemorethan
18hoursaftertheblackout.Rollingblackoutscontinuedtoaffectabout5%ofthepopulation
onthenexttwodays(2930September)astheelectricitycompany,ENEL,continueditseffort
torestoresupply.
Lessons learned
TheUCTEreviewmadeelevenrecommendations,however,onlyafewmightbeapplicable
totheNEM:
Nationalregulationsshould,insofarastheyarenotyetimplemented,provide
for:
Bindingdefenseplanswithfrequencycoordinationbetweenload
shedding,ifany,andgeneratortripsettings;
Bindingrestorationplanswithunitssufficientlycapableoftriptohouse
load(TTHL)operationandblackstartcapability;
AsupporttoolfordynamicanalysisandmonitoringoftheUCTEsystemis
needed,so,theongoingWideAreaMeasurementSystem(WAMS)installation
programwasaccelerated;
Regionsshouldhavebindingdefenseplanswithfrequencycoordination
betweenloadshedding,ifany,andgeneratortripsettings;and
Onloadtapchangingtransformerblockingincaseofseverevoltagedrop
shouldbeacceptedpractice.
Aswithotherblackouts,therewasaperiodofconfusionwhilethesituationwas
understood,interconnectionswereusedfreelyasavailableaspartoftherestoringthe
system,andsomeblackstartandTTHLgeneratorsfailedtoperform.HavingalltheTTHL
generationavailablewouldnothavechangedtheoverallplanrestoringinterconnections
wouldremainthefirststephowever,restorationwouldhavebeenbothquickerand
smoother.IthasbeenestimatedthatnearlyalltheItalianload(exceptSicily)couldhave
beenrestoredin12hoursratherthan18.
Page 40 of 69
DGAConsulting
2011 San Diego, US

Ontheafternoonof8September2011,an11minutesystemdisturbanceoccurredinthe
PacificSouthwest,leadingtocascadingoutagesandleavingapproximately2.7million
customerswithoutpower.16TheoutagesaffectedpartsofArizona,SouthernCalifornia,and
BajaCalifornia,Mexico.AlloftheSanDiegoarealostpower,withnearlyoneandahalf
millioncustomerslosingpowersomeforupto12hours.Thedisturbanceoccurrednear
rushhour,onabusinessday,snarlingtrafficforhours.Schoolsandbusinessesclosed,some
flightsandpublictransportationweredisrupted,waterandsewagepumpingstationslost
power,andbeacheswereclosedduetosewagespills.Millionswentwithoutair
conditioningonahotday.

September8,2011,wasarelativelynormal,hotdayinArizona,SouthernCalifornia,and
BajaCalifornia,Mexico,withheavypowerimportsintoSouthernCaliforniafromArizona.
Infact,importsintoSouthernCaliforniawereapproximately2,750MW,justbelowthe
importlimitof2,850MW.(Septemberisgenerallyconsideredashoulderseason,when
demandislowerthanpeakseasonsandgenerationandtransmissionmaintenanceoutages
arescheduled.)
DespiteSeptemberbeingconsideredashouldermonth,temperaturesintheImperialValley
(farsoutheasternCaliforniadeserteastofSanDiego)reached46C.Theloadoftheutility
servingtheImperialValleyareaheadedtowardnearpeaklevelsofmorethan900MW,
requiringittodispatchlocalcombustionturbinegenerationaccordingtoestablished
operatingprocedures.Fortyfourminutesbeforetheinitialeventthelocalutilityoperator
didnotnoticethatthelossofonekeytransformerwouldoverloadasecondtransformer
aboveitstrippoint.Theloadingonthesetransformerswaspivotaltothisevent.
16. AdaptedfromtheFERC/NERCStaffReportontheSeptember8,2011Blackout,April2012.
Page 41 of 69
DGAConsulting
ThesupplyanddemandconditionsofSanDiego
CountybeforetheblackoutareshowninTable4.
(Notethatthisisonlypartoftheareaaffectedby
Table 4: San Diego

County pre-event
supply and demand
MW
theblackout.)
Supply
Theimportantelementsofthesystemareshownin
Power plants
Figure15.ThecentralSanDiegocentralbusiness
Mexico
districtisintheareaaroundtheOldTownand
2,229
1,809
420
Imports
2,657
Missionsubstationsshowninthefigure.Thefigure
From north
1,287
alsoshowsthetwocriticalimportpathsintothe
From east
1,370
area500kVfromtheeastand230kVfromthe
Demand
4,293
north.TheSanOnofrenuclearplantislabeled
SONGSinthefigure.
Figure 15: San Diego area map of event
The blackout
Atabout14:00therewereproblemsatthe500kVNorthGilasubstationinArizona.An
experiencedtechnicianinadvertentlyskippedtwoofthesixteenstepsnecessarytoresolve
theproblem.Thisresultedinarcing,leadingtoaphasetophasefaultthattrippedthe
SWPLNorthGila500kVline.
Page 42 of 69
DGAConsulting
Arizonautilityoperatorserroneouslybelievedthattheycouldreturnthelinetoservicein
approximately15minutes,becausetheyhadnosituationalawarenessofalargephaseangle
differencecausedbytheoutage.Moretimewouldhavebeenneededtoredispatch
generationtoreducethephaseangledifferencetotheallowedvalue.
Asaresultofthelinetrip,flowsredistributedacrosstheremaininglinesintotheSanDiego,
ImperialValley,andYumaareas.ImmediatelyafterthetriptransformersatImperialValley
loadedto118%oftheiremergencyratings.Bothtransformerstrippedin40seconds.About
aminutelater,allthetransmissionwithflowsfromArizonatrippedandsomefrom
northernCalifornia.
ThemajorityoftheflowdivertedtothenorthernentrytotheSanDiegoarea,Path44.Flow
onPath44increasedbyapproximately84%,from1,293MWto2,362MW.
ThesituationatthistimeisshowninFigure16
Figure 16: The southern California situation at 15:28
AutomaticdistributionundervoltageprotectionintheImperialValleyareasystembegan
trippingdistributionfeedersandsheddingload.From15:32:11to15:33:46,444MWofthis
loadtripped.Generatorsandothertransmissionlinesintheareaalsostartedtrippingby
about15:35.ThesituationatthistimeisshowninFigure17.
Page 43 of 69
DGAConsulting
Figure 17: The southern California situation at 15:32
Lookingatloadingfromthenorth,aggregateloadingonthesouthofSONGS230kV
transmissionlinesincreasedfromapproximately6,700ampstoashighas7,800amps.(There
isaSONGSseparationschemethatactivatesat8,000amps.)Theloadingsettledaround7,200
ampsandremainedthere.
Between15:35and15:37,additionaltransmissiontrippedintheeasternpartofthearea.The
aggregateloadingsouthofSONGSincreasedfromapproximately7,200ampsto
approximately7,800amps.WhenthelastlinetrippedaggregatecurrentonPath44
increasedto8,400amps,wellabovethetrippointof8,000amps.
Bythistime,thesouthofSONGSlinesweretheSanDiegoareas(includingImperialValley
andMexicosnorthernBajapeninsula)onlysourceofcriticalimportedgeneration.Ifthe
aggregatecurrentwasbroughtbelow8,000amps,theblackoutcouldhavebeenavoided,but
atthispointnooperatoractioncouldhaveoccurredquicklyenough.Hadtherebeenformal
operatingproceduresthatrecognizedtheneedtopromptlyshedloadastheaggregate
currentapproached8,000,andhadoperatorsbeentrainedonthe8,000ampsetpoint,itis
possiblethatoperationoftheSONGSseparationschemecouldhavebeenavertedbyearlier
controlactions.
Page 44 of 69
DGAConsulting
MillisecondsaftertheImperialValleylineoutages;severalsmallergeneratorstotaling
432MWtripped,pushingthesouthofSONGSflowto9,500amps.
At15:38,notquite11minutesaftertheinitiallinetripped,theSONGSseparationscheme
operated,effectivelyseparatingallfivesouthofSONGS230KVtransmissionlinesand
separatedtheentireSanDiegoareafromtherestoftheWesternInterconnection.
TheelectricalislandcreatedbyoperationoftheSONGSseparationschemeleftasignificant
imbalancebetweengenerationandload.Asaresult,thefrequencyintheislandrapidly
declined.Inlessthanasecond,theUFLSprogramsbeganactivatingwithintheisland.This
ledtothefinalcascadinglossofgenerationandloadinthearea.
ThefinalblackouttotalsareshowninTable5.
Table 5: San Diego blackout statistics

Generation
lost (MW)
Demand
interrupted (MW)
Customers
affected
San Diego Gas & Electric
2229
4293
1,387,336
Southern California Edison
2428
117*
CFE (Mexico)
1915
2205
1,157,000
333
929
144,000
76
389
69,694
74
18,000
6982
7890
2,776,147
Company
Comision Federal de Electricidad

Imperial Irrigation District
Arizona Public Service
Western Area Power Association
Total
* These customers are served via San Diego Gas & Electric facilities
Restoration
Noneoftheaffectedentitiesneededtoimplementblackstartplansbecausetheyallwere
abletoaccesssourcesofpowerfromtheirownoraneighborssystemthatwasstill
energized.Therestorationprocessgenerallyproceededasexpected,andsomeentities
restoredloadmorequicklythantheyhadexpected.Table6showshowlongittookthe
affectedentitiestofullyrestoretheirlostload,generation,andtransmission.Table7isa
similartableofgenerationrestorationtimes.
Page 45 of 69
DGAConsulting
Table 6: San Diego area load restoration times

Load lost
(MW)
Time until
demand fully
restored
Date
restored
Demand fully
restored (hrs)
4,293
03:23
9/9
12.0
2,205
01:37
9/9
10.0
929
21:40
9/8
6.0
389
21:12
9/8
6.0
74
22:23
9/8
6.5
Entity
Western Area Power Association
Table 7: San Diego area generation restoration times

Generation restored
Generation
lost (MW)
Time
Date
Hours
2,428
06:33
9/12
87
2,229
06:20
9/10
39
CFE (Mexico)
1,915
23:43
9/10
56
333
20:42
9/8
76
20:37
9/8
Entity
Comision Federal de Electricidad

Lessons learned
Itispossibletocompletelyrestorelargeamountsofloadusingonlyinterconnections.
TheofficialreviewbyFERCandNERChadonlyafewimportantrecommendations:
Theyfelttheregionalcoordinator(WECC)couldhavetakenamoreactiverole
incoordinatingtherestorationefforts.Theregionalcoordinatorhasthe
largestareaofvisibilityandmoreadvancedrealtimestudytoolsthanthe
transmissionoperators.Duringamultisystemrestoration,issuesarelikely
toarisebetweenneighboringsystemsthatmayrequireeitheraneutral
decisionmaker,orrapidtechnicalanalysisofunplannedsystemconditions.
Therewasa30minutedebatebetweenSouthernCaliforniaEdisonandSan
DiegoGas&ElectricregardingresettingtheSONGSseparationscheme
lockoutrelay.Thisdelayedrestoringthetransmissionpathfromthenorth.
Page 46 of 69
DGAConsulting
TASK 2INTERNATIONAL COMPARISON OF

REGULATORY ARRANGEMENTS TO PREVENT OR
MITIGATE BLACKOUTS INCLUDING RESTORATION
TheprimarypurposeofthisTaskistoprovideinternationalcomparisonstotheNEMSRS.
EachexampledescribestheregulatoryarrangementsrelatedtotheSRS.Totheextent
possible,thereportidentifiesanydocumentsthatestablishsimilarrequirementstotheSRS.
Thechapterdescribesfiveinternationalexamplesoftheregulatoryarrangementsdesigned
topreventoramelioratemajorsupplydisruptionsintheelectricitysystem.Theexamples
wereselectedinconsultationwithAEMCReliabilityPanelfromalistofsuggestionsbyDGA.
Theselectedsystemsarefromfourdifferentcountries,andthreecontinents.
Thefiveselectedwere:
PJM,US;
SouthAfrica;
Italy;
ERCOT,US;and
Ireland
TheAEMCReliabilityPanelincludedspecificitemstocompareintheirRFP.Eachoftheseis
summarizedinTable8,below.
Context for comparison

TheAEMCReliabilityPanelisinterestedincomparinginternationalregulationand
proceduresrelatedtoblackoutsandblackstartgeneration.SomeoftheissuesthattheAEMC
isinterestedincomparingincludetheminimumblackstartrequirementsregarding
amountsandresponsetimes.TheseareincomparisonwiththeSRSthatrequirestheAEMO
toprocureenoughblackstartcapacitytoenergizeenoughgenerationtosupply40%ofthe
loadwithin4hours.
Page 47 of 69
DGAConsulting
Table 8: International comparison of regulatory arrangements to prevent or ameliorate blackouts

System/region
Characteristic
PJM (page 50)
South Africa (pg 53)
Italy (page 53)
ERCOT (page 54)
Ireland (page 56)
Any specific variables for system

restoration (and their values), such as time
for restoration of auxiliaries or generation,
specific volume of generation capability to
be restored, or reliability requirements for
restart units
Synchronize in 3
hours
4 hours to energize
nuclear units
Each black-start plant

must energize a large
coal plant in 4 hours
None, only general

comments
Unwritten
understanding of 4
hours for nuclear
plants
Thermal black-start
plants must
synchronize within 30
minutes of being
energized
Any requirement for the system to be able

to restore certain quantities or percentages
of total load within a specific timeline
Time only for nuclear

units
Restore critical load
None
None
No time required
Amount is confidential
None
There are priority
loads
Any specification of the scale of the power

system event that must be addressed (such
as whether the event is assumed to be subregional, regional or countrywide), and how
this affects the level of restart services to
be procured
Assumes both zonal

and system-wide
blackouts, but the plan
is to use the zonal
black-start resources
to restore the system.
National
National
Entire system
National
Any deterministic requirements, such as a

minimum number of restart services that
must be procured (ie including any
contingency/reserve restart services)
At least 2 for each

zone
Can be located in
adjacent zone
2 plants for entire

nationa pumpedhydro plant and 2
large diesel at a coal
plant
None
None, but multiple

black-start units will be
needed for such a
large system
At least one blackstart unit in each of

four subsystems
Any system specific requirements, such as

a requirement to restore transmission
corridors to a stable voltage
None
All generators
>200 MW must be
capable of TTHL for 2
hours
None
None
None
Any restoration priority for specific loads

(eg sensitive economic loads such as
aluminum smelters, critical services such
Nuclear plants
Critical natural-gas
infrastructure
Nuclear plant
Hospitals, etc. all have

emergency generators
Large industrial loads
Nuclear plants
Natural-gas pipeline
compressors and
None
Page 48 of 69
DGAConsulting
System/region
Characteristic
as hospitals) or specific generators (such
as nuclear power stations), including
whether the system operator must prioritize
these generators when restarting the power
system
PJM (page 50)

Specific loads may be
designated by zone
operators
South Africa (pg 53)
Italy (page 53)

also have local
generation to supply
their critical loads
ERCOT (page 54)

processing stations
Ireland (page 56)
Any specific assumptions regarding the

underlying condition of the power system,
such as specific network or generator
outages
Generally normal
conditions (see text)
None
Peak load
Summer peak
Peak and off-peak
Any requirements related to the diversity

(eg fuel, strategic, electrical or geographic
location) of restart services
None
None
None
None
None
Page 49 of 69
DGAConsulting
TheeightcharacteristicsinTable8cansimplifiedasfollows:
1. Specificsystemrestorationvariablestheonlyrequirementsweretore
energizethesystembusofnucleargeneratorswithin4hours;
2. Loadrestorationamountsandtimingnoneofthesystemshadsuch
requirements;
3. Assumedblackoutscalevariedamongthesystems;
4. Deterministicrequirementsmostrequiremultiplesources,butallowed
usingtheinterconnections;
5. Systemspecificrequirementsgenerallynone;
6. Restoringpriorityloadsnuclearplantsandnaturalgaspumpingstations;
7. Underlyingsystemconditionassumptionsnormalconditions;and
8. Diversityrequirementsnone.
Caseswheretheresponsesshowninthetablearenontrivialarediscussedbelow.
PJM, US
ThePJMInterconnectionisaregionaltransmissionorganization(RTO)thatcoordinatesthe
movementofwholesaleelectricityinallorpartsof13statesandtheDistrictofColumbia.
Actingasaneutral,independentparty,PJMoperatesacompetitivewholesaleelectricity
marketandmanagesthehighvoltageelectricitygridtoensurereliabilityformorethan61
millionpeople.PJMincludesthe21transmissionzonesshowninFigure18with184,000MW
ofgeneration.Eachzoneisresponsibleforacquiringthenecessaryblackstartcapacity.17
17. PJMsblackstartrequirementsaredescribedinPJMManual12:BalancingOperationsandManual36:System
RestorationAttachmentA:MinimumCriticalBlackStartRequirement.Themanualsareavailableat
www.pjm.com/documents/manuals.aspx.
Page 50 of 69
Figure 18:
PJMs
DGAConsulting
21 transmission zones
Specific system restoration variables

Blackstartgenerationisintendedtoenergizeunitswithahotstarttimeoffourhoursor
less.Blackstartgeneratingunitsmustbeabletoclosetheoutputbreakertoadeadbus
withinthreehoursofarequestfromthelocaltransmissionownerorPJM.PJMmayrequire
someblackstartresourcestoadheretolessthanathreehourstarttime.(Athreehourstart
timemaynotbeappropriatetomeetnuclearpoweroffsitesafeshutdownloadrestoration
requirements.)
Theremustbeenoughblackstartgenerationineachzonetobeabletostart110%ofthe
criticalload.ThetransmissionoperatorineachPJMzonedeterminesthecriticalload.
Criticalloadsinclude:
Crankingpowertocriticalgeneration(thermalunitswithhotstarttimesless
than4hours);
Crankingpowertocombustionturbines;
Page 51 of 69
DGAConsulting
Offsitenuclearstationlightandpower(2independentfeeds);
Includingunitsofflinepriortodisturbance;
Ensureasafeshutdownofnuclearorothergeneration;
Facilitateastationstartup;
Unitswithahotstarttimeof4hoursorless;
Criticalgasinfrastructure(keyinaquickrestart);
Powertoelectricinfrastructure;
Lightandpowertocriticalsubstations;
Pumpingplantsforundergroundcablesystems;
Criticalcommunicationequipment*;
Criticalcommandandcontrolfacilities*;and
Underfrequencyloadshedcircuits*.
(Itemsmarkedwithanasterisk(*),canbesuppliedbylocalstorageorgeneration.)
Underlying system condition assumptions

ThefollowingassumptionsareappliedtoplanningforaSystemRestoration:
Totalzoneblackout(noassistancefromexternalsystems);
Normalweatherpattern(notaresultofanaturaldisasterorextreme
weather);
Intermediatetopeakloadlevel(marginalsteamunitshot);
Minimalequipmentdamage(transmission/generation);
Normalworkinghours(sufficientpersonnellocatedinthefieldoroncall);
Variablessuchasthecurrentschedulingstrategies,theamountofnuclear
unitsoperating,loadlevels,weatherconditions,equipmentdamageandthe
amountofdirectpurchasesmayimpactrestorationtimes.Longerrestoration
timesmayresultfromdisturbancesduringoffpeakhoursordisturbances
resultingfromextremeweatherpatterns.Fasterrestorationtimesmaybe
possibledependentuponactualsystemseparationboundaries,theabilityto
importgenerationandstatusofequipment.
Otherhighpriorityloadwhichshouldbeconsideredearlyintherestoration
processinclude:
Page 52 of 69
DGAConsulting
Crankingpowertogenerationwithgreaterthan4hourstarttime;
Powertoelectricinfrastructureinaccordancewithtimeframedefinedin
restorationmanual;
LightandPowertorestorecriticalsubstations(ifapplicable);
Pumpingplantsforundergroundcablesystems;
CriticalCommunicationEquipment;
Criticalcommandandcontrolfacilities;and
Underfrequencyloadshedcircuits.
South Africa
TheSouthAfricansystemreflectsmanycharacteristicsofacentralizedsystem.Blackstart
requirementsaresetintheGridCode.TheCoderequirestwoblackstartplantsforthe
nation.Oneisa4x250MWpumpedhydroplantandtheotheristwo2030MWdiesels
locatedatalargecoalplant.
TheCodealsorequiresallunits>200MWtobeabletotriptohouseload(TTHL)foratleast
twohours.Interestingly,theseunitsarenotcountedaspartoftheblackstartrequirements.
Theemphasisisonstartingthelargecoalunits.
Italy
Italianmapsandotherdatawerediscussedaboveregardingthe2003blackoutbeginningon
page30.RecentItalianpeakloadwasabout51,000MW,thoughthealltimepeakofalmost
57,000MWoccurredin2007.
TheTernaGroupisthegridoperatorfortheItalianelectrictransmissionsystem.Ternais
thelargestindependentTransmissionSystemOperator(TSO)inEurope.Itownsthe
NationalHighVoltageTransmissionGrid,andisresponsibleforthetransmissionand
dispatchoftheelectricityfortheentireCountrywith~3,500employees.Itoperatesabout
63,900kmofthreephaseconductors,21interconnections,and491substations.
Page 53 of 69
DGAConsulting
TheItalianrestorationplanisapublicdocument,butitisallinItalian.18Therearetwo
prioritiesintheItalianrestorationplan:
1. RestorecorridorsfromtherestofEuropeacrosstheAlpstoquicklyenergize
portionsoftheinternalgrid,and
2. Emphasizerealtestingofrestorationstrategiesandtrainingoftheoperators.
TheblackstartstrategycombinesrestorationwithsupplyfromtherestofEurope(1st
priority)andcreatingelectricislands,withtheislandsprogressivelymeshedand
synchronizedwiththerestofthesystem.
WhilethereisnomarketforblackstartserviceinItaly,anumberofgeneratingunitsare
designatedasblackstartand,assuch,areboundbyagreementswithTerna(e.g.theyhave
toagreeontheirmaintenanceperiod,aresubjecttoperiodictestingofblackstartcapability,
etc.).
TheunitswithblackstartcapacityareidentifiedinstudiesperformedbyTerna.Thekey
variablesaregeographicallocation,transmissiongridtopology,andthepossiblerestoration
transmissioncorridors.Theselectedblackstartunitsaredefinedandtestedthrough
dynamicsimulationsandrealtimetesting,wheneverpossible.
ERCOT, US
TheElectricReliabilityCouncilofTexas(ERCOT)managestheflowofelectricpowerto24
millionTexascustomersrepresentingabout90percentofthestateselectricload(see
Figure19).Astheindependentsystemoperatorfortheregion,ERCOTschedulespoweron
anelectricgridthatconnectsmorethan70,000kmoftransmissionlinesand550generating
units.Thesystempeakloadisabout70,000MW.Therearemorethan1,400activeentities
thatgenerate,move,buy,sellorusewholesaleelectricity.
18. PianodiRiaccensionedelSistemaElettricoNazionale,(RestorationPlanoftheNationalElectricSystem).
Page 54 of 69
Figure 19:
ERCOT
DGAConsulting
area of Texas
InERCOT,blackstartserviceisanancillaryserviceprovidedbyaresourceabletostart
withoutsupportoftheERCOTtransmissiongrid.Itisprocuredcompetitivelyeverytwo
years.ERCOTisallocatedaspecificamountoffundingtospendtoprocureblackstart
services.
Factorsusedinselectingblackstartunitsinclude:
Fuelsupply(notdiversity),capabilitytorunonauxiliaryfuelsupply;
Locationrelativetomajorloadcenterortransmissioncorridorthatconnects
nextstartresources;
Startuptime,ramprates;and
Resourcecost.
Page 55 of 69
DGAConsulting
Ireland
Irelandis,literally,anisland.TherearetwoHVDCinterconnectionswiththeUK500MWto
theDublinareafromWalesand500MWtotheBelfastareafromScotland.Otherwise,they
dependontheirownislandresourcestosupplyload.Itshouldalsobenoted,thattheHVDC
linksdonothaveblackstartcapability.Peakloadontheislandisabout6,500MWandis
suppliedbyalmost10,000MWofgeneration.Inaddition,windgenerationismorethan
2,600MW,andisexpectedtotriple.
Restoring priority loads

Priorityloadsincludethepowersystemcontrolcentersthemselves,hospitals,airportsand
otherloadsofnationalimportance.Ingeneral,Irelanddoesnothavethekindofpriority
loadsthatAustraliahas;e.g.smelters.
Underlying system condition assumptions

Irelandcanhaveanationalblackoutatanytime,dayornight,soavarietyofloadconditions
arestudied.
Page 56 of 69
DGAConsulting
TASK 3CHANGES ANTICIPATED FOR BLACK-START

PLANNING IN SYSTEMS WITH VERY HIGH LEVELS
OF RENEWABLE GENERATION
Afewregionsintheworldareexperiencingtheeffectofveryhighpenetrationlevelsof
renewablegeneration.TheseincludeHawaii,Ireland,Denmark,andSouthAustralia,
amongothers.Thesesystemswillfacesomenewchallengesastheproportionofrenewable
energygenerationincreasesfurther.Thischapteraddressessomeofthesechallengesand
potentialmitigationmeasures,especiallyastheyapplytotheNEM.
Nature of renewable generation affecting blackouts and black-start

Thereisarangeofrenewableenergygenerationconsideredfromwindandphotovoltaic
(PV)torenewablesthatincludehydro,geothermal,wasteheat,etc.Thelattercanbe
dispatchedlikeconventionalgenerationthatis,theiroutputcanbecontrolledupordown
andtheydonotrelyonvariableenergysourceslikethewindorsunshine.Thischapter
focusesonthetwotypesofrenewableenergygenerationthatareproblematicforsystem
operationwindandsolar.
Windisusuallyeasiertointegratethansolar.Windblowsdayornight,thoughonshore
windtendstovarymoreandoftenstopsduringveryhotweather.Offshorewind,onthe
otherhand,tendstoblowmoresteadilydayandnight.Inaddition,windgeneratorshave
somerotationalinertiaintheirrotatingbladesthatreducestheirinstantaneousvariability.
Solargeneration,incontrast,onlyproducesenergyduringthedayfromafewhoursafter
sunrisetoafewhoursbeforesunset.Solarisalsosubjecttoveryrapidoutputchangesas
cloudspassoverhead.
Anotherimportantdifferenceisinprojectsize.WindgenerationisMWscaleunitsthatare
oftenpartofawindfarm,whilerooftopPViskWscaleinindividualunits.(Therearealso
largescalesolarprojectsthatareMWscale.)Thisisanimportantdifference.
RooftopPVisusuallyconnectedatlowvoltagesathomesorbusinesses.Thesesmall
projectsoperatebasedonlocalcontrolandareinvisibletosystemoperators.Whena
distributionfeederhasmanyrooftopPVsystems,itcanproduceenoughpowertobackfeed
powertothesubstation.
Page 57 of 69
DGAConsulting
TheMWscaleprojectswillusuallybeconnectedathighervoltagesandcanbemonitoredin
realtimebysystemoperators.Furthermore,windisusuallyproducedinwindfarmsthat
includemultipleunitswithasingleconnectionpointthatincludesrealtimemonitoring.19
Bothwindandsolargeneratorsproducedirectcurrent(DC).Bothuseaninverterto
transformtheDCintoalternatingcurrent(AC)power.Thesecharacteristicsprovideatleast
twoadvantages:
1. TheycanchargebatteriesdirectlyavoidingAC/DCtransformationlosses;
and
2. Theinvertersoffersomepotentiallyusefulfeaturesveryrapidcontrol,and,
advanced(fourquadrant)inverterscancontrolbothpower(MW)andvoltage
(var).
Theimpactofthesefactorsonblackoutsandsystemrestorationisdiscussedbelow.
Impact on blackouts
WindandsolarPVgenerationcanhavesomepositivebenefits,buttheycancauseserious
operatingproblems.Evenso,therearemitigationmeasuresthatcanbeusefulinpreventing
blackoutsorassistingrestoration.
System inertia
Inconventionalgeneration,rotatinginertiaprovidesarapidreserveofpowerthathelps
stabilizesystemfrequency.Whenasystemlosesgenerationthefrequencydeclines.The
rotatinginertiaofconventionalgenerationinstantlyincreasesgeneratoroutputastheyslow.
Duringtheseconditionsgeneratorscanbrieflyprovidemuchmorethanratedoutput.After
afewsecondsgovernorandothercontrolswillincreasetheenergy(fromsteamornatural
gas)deliveredtothegenerator,increasingitsoutput.
Sincethefinalstagesofblackoutsusuallyoccurinlessthanaminutefasterthanhumans
canunderstandandresponditisautomaticactionsthatpreventblackouts.Oneoftheseis
theinertialresponseofgenerators.Thisslowsthespeedthatfrequencyfalls,givingother
automaticsystemstimetorespond.(Thishappenedinalltheblackoutexamplescitedas
partofTask1.)
19. AllthewindgenerationinSouthAustraliaisinwindfarmsofatleast30MW.
Page 58 of 69
DGAConsulting
Inmajorpowershortages,UFLSandundervoltageloadshedding(UVLS)aredesignedto
arrestthefrequencydeclineandgivethegeneratorgovernorstimetoincreasegeneration
enoughtopreventatotalcollapse.Allofthisisprettyautomatic.Intheminutesandhours
thatfollow,systemoperatorsarethenabletorestorenormaloperation.
RooftopPVgenerationhasnorotationalinertiaandwindhasverylittle.So,assystem
frequencydeclinestheycontinueproducingthesameoutput.RooftopPVsystemsalsodo
notrespondtovoltagevariations.(MoreonthisunderPossiblemitigationmeasures
startingonpage64.)
Inaddition,invertersusedwithwindandPVareusuallysizedatabout105%ofthesources
maximumratedoutput,providinglittleroomforshortburstsofpowerlikeconventional
generation.Conventionalgenerators,incontrast,provideveryhighfaultcurrents.While
thismightseemtobeabadthing,itisimportantinprotectingthesystem.Protectiverelays
andfusesaresettorespondwhenfaultcurrentsoccurthataremuchhigherthaninnormal
operation.Conventionalgeneratorswillbrieflyprovide400%ormoreoftheirratedoutput
underfaultconditions.Thisbriefsurgeofcurrentiswhatsignalsrelaystooperateand
causesfusestoblow.Withoutsuchhighfaultcurrentfromconventionalgeneratorsor
someothersource,protectionplansandsettingswillhavetobecompletelyrevised.
AstheamountsofwindandPVincreaseduringoperation,therewillbelessconventional
generationoperatingaspartofnormaleconomicdispatch.20Thiswillsignificantlyreduce
therotatinginertiaofthesystem.Itwillalsoreducefaultcurrentlevelsthatwilldisrupt
systemprotectionschemes.Theresultisthatmuchmorerapidfrequencyswingsoccur
duringcontingencies.Thus,automaticsystemsdesignedtopreventablackoutwillhave
muchlesstimeact.
Ramp-rates (rate of change of frequency)

Besidesinertialresponsediscussedabove,conventionalgeneratorscanincreaseand
decreasetheiroutputoveraperiodofminutesautomaticallyorunderoperatorcontrol.Itis
generatorgovernorsthatvarytheenergyinputtotheturbinesthatproduceelectricity.
Alongwithanyinertialresponse,thesecontrolsareusedtomaintainanearlyconstant
20. WindandPVhavenearzeroshortrunmarginalcostsandwillbedispatchedaheadhighercostconventional
generation.
Page 59 of 69
DGAConsulting
frequencyascustomerloadandotherconditionschangefromsecondtosecondduringthe
day.ThisabilityismeasuredasramprateandusuallystatedasMW/minute.
Theamountoframprateforaunitdependsonitsizeandtechnology.Generally,larger
unitshavehigherramprates;andcombustionturbineshavehigherrampratethansteam
units.Thesystemoperatormustseethatthecombinedrampratesoftheoperating
generatorsareenoughtomeettheexpectedchangesinsystemload.
TheoutputofwindandPVoftenhassteeprampsasopposedtothecontrolled,gradual
rampupordowngenerallyexperiencedwithelectricitydemandandtheoutputof
conventionalgeneration.Managingtheserampscanbechallengingforsystemoperators,
particularlyifdownrampsoccurasdemandincreasesandviceversa.
TherecanberapidfluctuationsinoutputofrooftopPVsystemsfrompassingclouds.Ifthe
conventionalgeneratorscannotrampfastenoughtomatchthevariabilityofthePV(or
wind)systems,afrequencymismatchmayaffectthewholesystem.Thiscouldleadto
underfrequencyloadsheddingorevenblackouts.
Under-frequency and under-voltage response

Asmentionedabove,adistributionfeederwithalotofrooftopPVcanbackfeedthe
substationduringsunnydays.Thiscanbeachallengeinsettingprotectiverelaysandfuses
inthesefeeders.Inpreblackoutconditionsthisbackfeedcancauseseriousproblemswith
UFLSandUVLS.
NormallyUVLSandUFLSaredesignedtoshedloadwhenloadexceedsgenerationandeither
frequencyorvoltagesfalltoofarand/ortoofast.Reducingloadinsuchconditionsallows
generatorstoregainbalancewithloadandslowlyrestorenormalconditions.
IfafeederthatispartoftheUVLSorUFLSsystemisbackfeedingitwillhavethewrong
impact.Insteadofhelpingrestorethegenerationloadbalancebyreducingsystemloadit
willreducesystemgeneration,andmakethesituationworse.Theoppositeofwhatwas
intended.
Complicatingmattersfurther,isthatsuchabackfeedcanonlyoccurduringdaylight,and
typicallybetween10:00and15:00whensunshineisbrightest.Sosuchafeederwouldneed
tobeexcludedfromUFLSandUVLSduringthesedaylighthours,butotherwisebeincluded
normally.
Page 60 of 69
DGAConsulting
Asdiscussedabove,highlevelsofwindandsolarPVwillreducetheamountofconventional
generationoperating.Theresultingreducedrotatinginertiameansthatsystemfrequency
willfallfasterwhenageneratorislost.(Frequencywillalsoincreasefasterwhenalarge
loadislost.)Thismakesthesystemlessrobustandmoresusceptibletoblackouts.
ThereisarelatedissuethatisimportantwithwindandPVlowvoltageridethrough.
Mostinvertercontrolsaredesignedtodisconnectwhenvoltagesfalltoacertainamount.In
systemswithhighlevelsofwindandPV,voltagescanfallfasterandfartherthansystems
withmoreconventionalgeneration.Inthesesituations,voltagescanremainlowlonger,
and,ratherthandisconnect,thewindandPVshouldremainconnectedlongertoseeif
voltagerecovers.Otherwise,trippingthewindandPVduringpowershortageconditions
willmakeconditionsworse.
Coincident frequency reaction

ManyutilitiesestablishedpoliciesanddesignstandardsforwindandrooftopPVgeneration
beforeitwasclearhowmuchofthisgenerationcouldbeinstalledonthesystem.Windand
rooftopPVcommonlydisconnectwhenlosingsourcepowerandwaitafixedperiodto
restartoncepowerisrestored.Thiskeepsrepaircrewssafewhenrepairingdamagethat
causedalocaloutage.Thesesystemsalsodisconnectwhenthefrequencydropstoaset
levelorvoltagegetstoolow.
SinceallthesewindandPVsystemsrespondwiththesamesettings,theyarecoincident.
Theywillalldropoutataboutthesamefrequencyorvoltage.Thiscanproduceadditional
shockstothesystemduringamajordisturbanceincreasingthechanceofablackout.
Over voltages
HighlevelsofwindandrooftopPVcancausehighvoltagesinthetransmissionsystemand
ondistributionfeeders.Voltagestendtobehigheratgeneratingsourcesandthese
generatorsraisevoltages.Thiscanbeaspecialproblemondistributionfeederswithalotof
rooftopPV.Voltagescanbecomehighenoughthatthesubstationequipmentcannotkeep
voltageswithinacceptablelimits.
BoththeseresourcesuseinverterstoconverttheirDCoutputtoAConthesystem.These
invertersgenerallyusethesimplestandleastexpensivedesignsespeciallyforrooftopPV.
Theseinvertersonlycontrolthepoweroutputanddonotcontrolvoltageorvaroutput.
(Moreonthisbelow.)
Page 61 of 69
DGAConsulting
Restart capability
OneadvantageofwindandPVcomparedwithconventionalgenerationisthattheycanbe
restartedandbroughttofulloutputveryquicklyassumingthesunisshiningandthe
windisblowing.Itcantakeonlyaminuteortwoifconditionsareright.Mostinvertersare
designedasfrequencytakerstheymatchthesystemfrequencyandvoltageofthepower
system.Theyareabletomatchafairlywiderangeoffrequenciesawiderrangethan
conventionalgenerators.
Inverterscanbedesignedtoprovidetheirownfrequencysignal,independentoftheoverall
system.WindorPVgenerators,especiallythosewithassociatedenergystorage,couldbe
usedtorestartafterblackoutandformanelectricalisland.Suchanarrangementcould
speedrestorationandaddconsiderableflexibility.
Impact on restoration
HighamountsofwindandrooftopPVinasystemwillimpactsystemrestorationfollowinga
blackout.Oneoftheobviousimpactsisthatloweramountsofconventionalgenerationwill
bewarmoravailableforTTHL.Thisconventionalgenerationwouldbeamongthefirst
resourcesusedtorestartthesystem.Withmorerenewablegenerationtherewillbeless
conventionalgenerationoperating,makingrestorationharder.
Restoringthesystemafterablackoutinvolvesreenergizingthetransmissionsystemwhile
balancinggenerationandload,andcontrollingvoltages.Windgenerationhassome
advantagesoverrooftopPVinrestoration.Windgenerationisconnectedtothetransmission
system,monitoredinrealtimebythesystemoperator,and,withinlimits,canbedispatched.
Windforecastmodelscouldestimatetheamountofenergythatwindgenerationcan
producefollowingablackout.Thesewindgeneratorscanbeusedinrestoringthesystem
especiallyiftheirinvertershavevoltagecontrol.
RooftopPVdoesnothavethevisibilityorcontrolthatwindgenerationhas.ThesePV
systemsdisconnectwhenpowerislost.Aspowerisrestoredtodistributionfeeders,these
PVsystemswouldremainoff.Connectingthesefeederswouldaddtheexpectedtoload
usedtobalancegenerationintherestartprocess.However,afterasetdelay,thePVsystems
wouldautomaticallyspringbacktolifereducingthenetloadonthefeederand
complicatingrestoration.OnfeederswithalotofPVtheimpactwouldbeevenworse.
Ofcourse,theseproblemswithPVwouldonlyoccurduringthedaytime.Sosystem
operatorswouldneeddifferentproceduresfordayornightconditions.And,sinceamajor
Page 62 of 69
DGAConsulting
blackoutmighttake12hoursormoretofullyrestore,restorationwouldprobablyoccur
partiallyinthedayandpartiallyinthenight.
Possible mitigation measures

Therearesomereasonablemeasurestomitigatethenegativeimpactsofwindandrooftop
PV.Therearealsosomemoreextremeoptions.
Conventional generation
Obviouslyenoughconventionalgenerationwithblackstartcapabilitywouldallowa
successfulrestart.Mostsimplecyclecombustionturbinescanblackstart.Similarly,the
combustionturbinesectionsofcombinedcycleplantscanusuallyalsoprovideblackstart
capacity.
Increased inertia and fault currents

Systeminertiaandfaultcurrentarereducedasmorewindandsolargeneration(andless
conventionalgeneration)operates.Onesolutionistousesynchronouscondensersto
increaserotatinginertiaandfaultcurrents.Synchronouscondensersarephysicaldevices
connectedtothepowersystemthatrotateatsynchronousspeed.Theactualdeviceisvery
similartotheelectricalportionofanelectricgenerator.Itincludesarotor,statorandexciter
justlikeagenerator.Thebigdifferenceisthattheydonothaveanythingthatprovides
mechanicalpowertoproduceMW.Generallyitisthemechanicalpowersourcethatisbyfar
thelargestandmostcomplexpartofageneratingplant.Asynchronouscondenserisa
muchsmallerandsimplerdevice.Theyproviderotatinginertia(thoughmuchlessthana
completegenerator)andtheycanbrieflyprovidefaultcurrentofabout400%ofrated
output.
Itisalsopossibletoprovidevirtualorsyntheticinertiausingelectricalstoragewith
inverters.Thesedevicescanprovidepowertohelpstabilizesystemfrequency.They
operatecontinuously,brieflysupplyingandabsorbingpowertomoderatefrequency
fluctuations.Duringmoreextremefrequencyexcursions,theycanprovidetheirfulloutput.
UVLS
and UFLS systems
TheUVLSandUFLSsystemsneedtobereviewedandrethought.Thefasterfrequencydrop
withhighlevelsofwindandPV,requiresreevaluatingsettingsforUVLSandUFLS.In
general,morecustomerloadwillneedtobeshedtocontrolthefrequencydrop.More
feederswillneedtobecontrolledbytheUFLSandUVLSsystemsandtheywillprobablyneed
Page 63 of 69
DGAConsulting
moresteps.Thesestepswillneedtostartatahigherfrequencyandtripmoreloadto
effectivelystopthefrequencydeclineorvoltagecollapse.
TheUVLSandUFLSsystemsalsoneedtobeadjustedfordayandnightoperation.Feeders
withalotofrooftopPVwillneeddifferentsettingsfordayandnighttimes.Thismay
requiredifferenthardware,communicationandcontrolsystemsthannowused.
PresentpracticeandregulationssetthestandardsforconnectingrooftopPV.Theseinclude
requirementsaboutdisconnectingwhenfeederpowerislostandthedelayinrestartingafter
powerisrestoredtothefeeder.ThecoincidentrestartofthesePVunitsisaproblem.A
flexiblerequirementthatallowsrooftopPVtousedifferentrestartdelayswouldreducethe
coincidenceproblem.
Inverter capability and design

Advancesininverterdesigncangivethemfourquadrantcontrol.Thismeansthattheycan
controlMWandvaroutput(allfourquadrants).Thisisabigadvantagebecausethese
inverterscanhelpcontrolvoltageeitherbyraisingorloweringtheirvaroutput.
ThesefourquadrantinvertersarebecomingthestandardfornewwindandlargePV
systemsintheUSandEurope.TheyarenotnowrequiredintheNEM.Theseinverterscost
somewhatmorethantheirsimplerbrethren,butoffervitalfunctionalityastheamountsof
windandPVbecomehigh.Theybecomeevenmorevaluablewhenassociatedwithenergy
storage(discussedbelow).
South Australia interconnections

SouthAustraliaisthepartoftheNEMthathasthemostwindandPV.Thestateis
interconnectedwithVictoriathrougha500/275kVACdoublecircuittransmissionlineat
HeywoodandanHVDCconnectiontoRedCliffs.TheHVDCconnectionislimitedtoabout
220MWand460MWfortheACconnection.(TheAEMOisintheprocessofincreasingtheAC
connectionlimitto650MW.)
TheSouthAustralialoadistypically1,000MWto1,500MW.Installedthermalgenerationis
about2,500MW.Thestatehasabout600MWofPVand1,500MWofwindwithseveral
hundredMWmoreunderconstruction.TherewillbemanyhourswhenthiswindandPV
willalmostcompletelyservetheloadwithnooperatingconventionalgeneration.We
understandthattherehavebeentimeswhentheSouthAustralialoadwasentirelyservedby
renewablegeneration.
Page 64 of 69
DGAConsulting
ThisisaveryvulnerableinterconnectioninthatthelossoftheACinterconnectionwould
likelycauseastatewideblackoutduringhourswhenwindandsolargenerationwashigh.
StrengtheningtheinterconnectionbetweenSouthAustraliaandVictoriacouldsignificantly
improvethissituation.
Energy storage
EnergystoragemaybethegreatwildcardregardingintegratingwindandPV.
Inexpensiveenergystoragewouldmitigatemostoftheissuesdiscussedabove.Itcouldbe
usedtocontrolrampratesofwindandPV,allowgreatervoltagecontrolandimprovefeeder
stability.
Regulatorscouldrequirestoragefornewwindgeneratorstocontrolrampratesandprovide
someblackstartcapability.Otherstudieshaveshownthat,aswindandsolaraccountfor
morethanabout15%ofpeakload,storageforrampratecontrolshouldberequired.21Other
systemshavefoundthatlimitingrampratesto10%ofratedoutputperminutewas
effective.
Energystoragewouldallowfourquadrantinverterstofunctionunderawiderangeof
conditions.Theycouldprovideramprateandvoltagecontrolasdiscussedabove.They
couldalsobeusedtoimprovestabilitybyquicklyvaryingtheiroutputundertransient
conditions.
Besidesdedicatedenergystorage,customerdeviceslikeelectricvehiclescanprovidemuch
ofthesamebenefits.Itisalsopossiblethatthetransmissionownercouldplaceenergy
storageatkeysubstationstoimprovesystemoperationundernormalandblackstart
conditions.
Ofcourse,thisdependsontheeconomicsofstorage.Technically,however,energystorage
wouldbeveryeffectiveinmitigatingtheimpactofhighwindandPVpenetrationsinthe
supplysystem.Properlydesignedinvertercontrolswouldallowstoragetobeusedas
blackstartcapacitytohelprestorethesystem.
21. P.J.Palermo,Chen,K.,Korinek,D.,SmallIslandExperiencePlanningtoIntegrateLargeAmountsof
WindandSolarGeneration,CigrInternationalSymposium,BestPracticeinTransmissionandDistributionin
aChangingEnvironment,Auckland,September2013
Page 65 of 69
DGAConsulting
Page 66 of 69
DGAConsulting
CONCLUSIONS
General conclusions regarding major blackouts
1. Outages
a. TransmissionversusgenerationcausestheblackoutsreviewedinTask1
wereallinitiatedbyunexpectedtransmissionevents.Atransmission
failureleadstoaveryrapidincreaseinloadingordeclineinvoltages
leadingtoaseriesofotherequipmenttrips.Theresultisasudden,
usuallylarge,uncontrolledcustomeroutage.
Incontrast,withagenerationshortagethereisusuallyatleastseveral
hoursofadvancewarningofanimpendingshortage.Theseresultin
controlledrotatingcustomeroutages.
b. Notatpeakloadnoneoftheeventsoccurredunderpeakload
conditions.Itiscommontostudypeakconditions,butthesystemisoften
morevulnerableduringoffpeakseasonswhengeneratingunitsarenot
dispatchedoronmaintenance.Therearealsousuallytransmission
maintenanceoutagesthathaveledtoerrorsthatcauseoutages.
c. Inalltheseblackoutsthereweremultiplecontingencies,beyondnormal
operatingandplanningcriteria.
2. Restoration
a. Situationalawarenessisanimportantfirststep.Insomecases,lackof
awarenesswasanimportantfactorthatdelayedrestoration.
b. Whereinterconnectionswereavailable(notHawaiiorSarawak)operators
usedthemearlyinrestoringthesystem.
c. Thereareusuallyelectricalislandsthatmaintainservicethroughthe
blackout.
d. Withwidespreadoutages
o
Usuallysomeequipmentfailsbeyondtheinitiatingcauses;and
Somesetbacksoccurduringrestoration,usuallyduetovoltagecontrol
problems.
Page 67 of 69
DGAConsulting
General conclusions regarding black-start requirements

1. Energizingpartsofthesystemwithin34hoursiscommon,butfully
restoringthesystemmaytake12hoursormore.
2. Noneofthesystemsrequireapercentageofloadtobereadytoberestored.
Somehavespecificcriticalloads,usuallynuclearpowerstationauxiliary
supplies,thatneedtoberestoredfirstandtobeenergizedin34hours.
3. Multipleblackstartresourcesshouldbeavailable,thoughtheycanbein
neighboringnetworks.
4. Therearefewspecificrequirementsforvoltagecontrol,though,obviously,
voltagesmustbewithinsafelimits.
5. Blackstartstudiesareusuallyconductedfornormalconditions.
6. Noneofthesystemsreviewedhere,considerfueldiversityinidentifying
blackstartgeneration.
Specific comments for the AEMC and NEM situation:

1. Inamajorblackouttherewillbequiteabitofinitialconfusion30minutes
toassessthesituationandforrestorationtobeginiscommon;
2. ThefourhourrequirementfortheSRSobjectiveasshowninFigure1onpage
4isconsistentwithinternationalexperiencewithmajorblackouts;
3. Internationalexperienceshowsthatfullyrestoringcustomerloadcantake12
hoursormorefollowingamajorblackout;
4. Practicingablackstartplan,asdemonstratedbyDetroitEdisonand
ConsumersEnergy,speedsrestoration;
a. Positioningpersonnel;
b. Openingbreakersforacleanstart;and
c. Understandingvoltageissuesinablackstart;
5. Notallplannedgenerationwilloperateasexpected;
6. Therewillusuallybesomeelectricalislandsthatremaininservice;
7. Allgenerators,thatcanhelpfollowingtheblackout,willhelpregardlessof
theSRASstatus;
Page 68 of 69
DGAConsulting
8. Controllinghighvoltagesontransmissionlineswillbeacommonproblem,
sooperatorsshouldhaverelevanttraining;
9. TheAEMOmusthaveclearauthoritytosettleanydisputesbetween
stakeholdersduringrestoration;
10. Transmissioninterconnectionsareusedearlyinrestoringthesystem.
11. AstudyofUFLSandUVLSisneededtodeterminetheproperamounts,
frequencysettings,andstepsizes;
12. AsurveyshouldbemadeoffeedersinSouthAustralia,andanyotherplace
withhighPVpenetration,toidentifythosewithsignificantamountsof
rooftopPV;
13. ConsiderrevisingconnectionstandardsforwindandrooftopPVinSouth
Australiaregarding:
a. Restartsettings,
b. Lowvoltageridethrough,and
c. Lowandhighfrequencyridethrough,
d. Storageforrampratecontrol(andblackstartsupport);
14. Consideraddingrequirementsforstorageassociatedwithlargewindfarms
andascommunitystorage;
15. AdditionalinterconnectionsbetweenSouthAustraliaandVictoriashouldbe
studied;
16. Asurveyshouldbemadeofcombustionturbinesandcombinedcyclegas
fueledunitsregardingtheirblackstartcapability;
17. Thereshouldbeadequatecommunicationandcontrolwithwindfarmsso
thatsystemoperatorscanmaintaincontrolforatleastfourhourswithout
externalpower;
18. DevelopmethodstoincludewindinrestorationplansforSouthAustralia;
and
19. WaystomonitorandincluderooftopPVinrestorationshouldbe
investigated.
Page 69 of 69

DGA Consulting International Comparison of Major Blackouts and Restorat

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

DGA Consulting International Comparison of Major Blackouts and Restorat

Caricato da

Copyright:

Formati disponibili

INTERNATIONAL

AEMC Reliability Panel International comparison of major blackouts and restoration

AEMC Reliability Panel International comparison of major blackouts and restoration

Developments in 2015 regarding SRAS and black start

AEMC Reliability Panel International comparison of major blackouts and restoration

AEMC Reliability Panel International comparison of major blackouts and restoration

TASK 1INTERNATIONAL COMPARISON OF MAJOR

Context for comparison

AEMC Reliability Panel International comparison of major blackouts and restoration

Figure 1: Three stages of restoration used by NEM

AEMC Reliability Panel International comparison of major blackouts and restoration

Table 1: International comparison of major supply disruptions

2008 Oahu, Hawaii

2011 San Diego

Major regional outage

Total system outage

Total island outage

2 days for most

Less than 30 minutes

Less than 30 minutes

The physical extent of the event (capacity

Human errors, if any, that triggered or

Protection equipment failures, if any, that

Errors at Bakun dam

The contribution of specific load or

Any unusual power system frequency,

Northern Ohio had

Time from the initial

Time for restart services to come online to

AEMC Reliability Panel International comparison of major blackouts and restoration

2008 Oahu, Hawaii

2011 San Diego

Any natural phenomena that contributed to

The readiness of generation and supply

Normal, but only 8 of

The readiness of network assets and the

Mostly normal, with

Any equipment damage and the extent to

The extent to which power system topology

Natural break points

The 100 km 275 kV

Italy is very long

There are only two

Whether designated restart services were

The system was reenergized solely from

Did the system restart plan operate as

There were problems

Yes, except for

Several failed restarts

Yes, but there was a

Also, some pockets

AEMC Reliability Panel International comparison of major blackouts and restoration

2008 Oahu, Hawaii

2011 San Diego

Any particular element of the restart

The economic cost of the event, such as

150 million USD

120 million USD

Any social impact attributable to the event

There was major

Chaos during rush

The cost of the restart services called upon

Any key lessons learned or recommendations arising from investigations of the

See text beginning at