Cogent Real-Time Systems Inc

RedundancyforOPC

byRobertMcIlvrideandAndewThomas
CogentRealTimeSystemsInc.

Earlyonemorning,MelFarnsworthwassittinginthecontrolboothattheHardyAutomotive
Partsassemblyline,drinkinghisfinalcupofcoffeebeforetheendoftheshift.Watchingthe
linemetergraph,henoticedthattheyieldandefficiencytrendsfortheLine3haddroppedto
zero.Helookeddownthroughthecontrolroomwindow,butLine3seemedtoberollingright
along.Whatwastheproblem?

Thelinewasrunningsmoothly,butMelwasntgettingthedataheneeded.Somewhere
betweenthePLCsandhisHMIdisplaytherewasadatadisconnect.Maybeitwasafieldbus
problem,orabadnetworkconnection.PerhapsitwascausedbyhisOPCserver,orpossibly
evenhisHMIsystem.Whateverthereason,sinceMelsdataconnectionwasasinglechain,
onebreakinthechainmeansthathedidntgethisdata.Tominimizethiskindofriskand
ensurethehighestpossibleavailability,missioncriticalsystemsoftenuseredundancy.

WhatisRedundancy?

Redundancyinaprocesscontrolsystemmeansthatsomeorallofthesystemisduplicated,or
redundant.Thegoalistoeliminate,asmuchaspossible,anysinglepointoffailure.Whena
pieceofequipmentoracommunicationlinkgoesdown,asimilaroridenticalcomponentis
readytotakeover.Therearethreetypesofredundantsystems,categorizedbyhowquicklya
replacement(orstandby)canbebroughtonline.Thesearecoldstandby,warmstandby,and
hotstandby.

Coldstandbyimpliesthattherewillbeasignificanttimedelayingettingthereplacement
systemupandrunning.Thehardwareandsoftwareareavailable,butmayhavetobebooted
upandloadedwiththeappropriatedata.Picturetheoldendaysofsteamlocomotives.The
coldstandbywastheextraengineintheroundhousethathadtobefiredupandbroughtinto
service.Coldstandbyisnotusuallyusedforcontrolsystemsunlessthedatachangesvery
infrequently.

Warmstandbyhasafasterresponsetime,becausethebackup(redundant)systemisalways
running,andregularlyupdatedwitharecentcopyofthedataset.Whenafailureoccurson
theprimarysystem,theredundantsystemcandisconnectfromthefailedsystemandconnect
insteadtothebackupsystem.Thisallowsthesystemtorecoverfairlyquickly(withinseconds,
usually),andcontinuethework.Somedatawillbelostduringthisdisconnect/reconnectcycle,
butwarmstandbycanbeanacceptablesolutionwheresomedatalosscanbetolerated.

Hotstandbymeansthatboththeprimaryandsecondarydatasystemsrunsimultaneously,and
bothareprovidingidenticaldatastreamstothedownstreamclient.Theunderlyingphysical
Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com

Cogent Real-Time Systems Inc

systemisthesame,butthetwodatasystemsuseseparatehardwaretoensurethatthereisno
singlepointoffailure.Whentheprimarysystemfails,theswitchovertothesecondarysystem
isintendedtobecompletelyseamless,orbumpless,withnodataloss.Hotstandbyisthe
bestchoiceforsystemsthatcannottoleratethedatalossofacoldorwarmstandbysystem.

ATypicalRedundantOPCSystem

WhatdoesredundancylooklikeinanOPCbasedsystem?Atypicalscenariowouldhavetwo
OPCserversconnectedeithertoasingledeviceorPLC,orpossiblyduplicatedevicesorPLCs.
ThosetwoOPCserverswouldthenconnecttosomekindofOPCredundancymanagement
softwarewhich,inturn,offersasingleconnectiontotheOPCclient,suchasanHMI.The
redundancymanagerisresponsibleforswitchingtothesecondaryOPCserverwhenany
problemariseswiththedatacomingfromtheprimaryOPCserver.Thisscenariocreatesa
redundantdatastreamfromthephysicalsystemallthewaytotheHMI.
OPC Data Access

is a group of
standards that
provides
specifications for
communicating
real-time data from
data acquisition
devices such as
PLCs to display
and interface
devices like
Human-Machine
Interfaces (HMI).

ThemostcommonuseofredundancyinOPCisforOPCDA,butitispossibletoconfigure
redundantOPCA&EorOPCUAsystems.Theprinciplesarethesame.Sometimes,onlarge
systems,itisnecessarytoconfiguremultipleredundantpairs.Redundancycanalsobe
configuredoveranetwork,usingDCOMorOPCtunneling.Foranetworkedconfiguration,the
redundancymanagerwouldnormallyresideontheOPCclientmachine,tominimizethe
numberofpotentialpointsoffailure.

Althoughcoldorwarmstandbymaybeusefulundersomecircumstances,typicallyanengineer
orsystemintegratorimplementingaredundantOPCsystemislookingforhotstandby.Thisis
themostusefulkindofredundancyinaprocesscontrolsystem,andatthesametimethemost
difficulttoachieve.LetslookalittlemorecloselyatthatallimportanttaskoftheOPC
redundancymanagerinahotstandbysystemmakingtheswitch.

Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com

Cogent Real-Time Systems Inc

MakingtheSwitch

Putsimply,ahotstandbyredundancymanagerreceivesdatafromtwoidenticalinputs,and
sendsasingleoutputtotheOPCclient.Itistheredundancymanagersjobtodetermineatall
timeswhichofthetwodatastreamsisthebest,andswitchfromonetotheotherassoonas
possiblewheneverthestatuschanges.Theswitchcanbetriggeredbyanumberofdifferent
kindsofevents:

Singlepointvaluechangetoorfromacertainvalue,achievingathreshold,etc.
Singlepointqualitychangeforexample,fromGoodtoanyotherOPCquality.
Multipleitemmonitoringifthequalityorvalueofanypointinagroupgoesbad.
Rateofchangemonitoringifpointschangevaluemoreslowlythanexpected.
Networkbreaksandtimeoutscheckedwithsomekindofheartbeatmechanism.

Oncetheswitchhasoccurred,thesystemortheredundancymanageritselfmighthavethe
abilitytosendanalarmoremailmessage,orevenlaunchsomekindofdiagnosticor
investigativeprogram.Itmightalsobeabletologdiagnosticinformationaboutthestateofthe
primaryOPCserverornetworkconnection.Andinasystemthatdistinguishesbetween
primaryandsecondaryinputs,therewilloftenbeameanstofavortheprimaryinput,and
switchbacktoitwhenpossible,sometimesreferredtoasafallback.

PracticalConsiderations

Theideaofredundancyisnotdifficulttograsp,butimplementingittakessomethought.An
initialdecisiononcold,warmorhotstandbywillimpactallaspectsoftheimplementation.The
choiceofproperhardwareandsoftwareiscriticalforawellfunctioningsystem.Robustsystem
architectureisalsoimportant,especiallyiftheconnectionisacrossanetwork.Inadditionto
selectingOPCserversandplanningthenetworkinfrastructure(ifnecessary),animportant
decisionwillbethesoftwareusedtomanagetheredundancy.Goodredundancymanagement
softwareshouldbeeasytouse,withnoprogrammingnecessary.Thetechnologyshouldbeup
todate,capableofrunningonthelatestversionofWindows.Thereshouldbeanabsolute
minimumchanceofdatalossduringaswitchover,evenoveranetwork.

TheTimerPitfall

Inpracticeitisnotpossibletoachieveacompletelyseamlessswitchoverinallcases,evenwith
ahotstandbysystem.Forexample,ifanetworkfailureoccursontheprimaryconnection,a
certainamountoftimewillpassbeforearedundancymanagercandetectthatfailure.Data
transmittedduringthisperiodwillfailtoarrive,buttheredundancymanagerwillnotbeableto
distinguishbetweenafailureandanormalpauseindataflow.

Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com

Cogent Real-Time Systems Inc

Manyredundancymanagersimplementtimerstoperiodicallycheckthenetworkconnection
statustotrytominimizethisdelay,butaswitchovermechanismbasedonperiodictimerswill
alwayssufferfromdataloss.Systemswithmultipletimingparameterswilloftenresultin
additivedelays,wherethefastestpossibleswitchoverforthesystemisthesumofthesetiming
delays.Inaddition,theuseoftimerstodetectnetworkfailurecanresultinaconfiguration
problemwherethesystemintegratormusttradeoffswitchoverlatencyagainstfalsepositive
networkfailuredetection.Thiseffectivelybecomesatradeoffbetweensystemstabilityand
responsiveness.

Usingtimerstoperiodicallycheckdatavaluesorqualities,orpolltheOPCservers,isalso
problematicbecausetimersintroduceunnecessarylatencyintothesystem.Whereasa
networkfailuremustbedetectedbasedontiming,adatavalueorqualitychangecanbe
detectedimmediatelyastheeventoccurs.Itisusuallybesttoavoidsystemsbasedontime
basedvaluechangedetection,anduseeventbasedobjectmonitoringinstead.

ObjectandLinkMonitoring

Agoodredundancymanagershouldbeabletosupportbothobjectmonitoringandlink
monitoring.Objectmonitoringmeanstheabilitytomonitorindividualpoints,andmakea
switchoverbasedonanevent.Forexample,ifadesignatedwatchdogtagchangesina
significantway,suchasturningnegativeorgoingoveraspecifiedthreshold,itcantriggera
switchtothesecondaryOPCserver.Ormaybeyoudliketomonitoragroupofpoints,andif
thequalityofanyofthemgoestoBadorUnconnected,youcanswitch.

Linkmonitoringisespeciallyusefulfornetworkedconnections.Yoursystemwillneedawayto
detectanetworkbreakveryquickly,topreventdataloss.Forhotstandbyonhighspeed
systemswithfastdataupdaterates,timeoutdetectionwithasubsecondresponserateis
essential.Inanyevent,thesystemshouldbeabletodetectatimeoutforafailednetwork
connection,aswellasafailuretoreceivedata.Thisdistinctionisimportant.Itmaytake
secondsorevenminutestodetectacommunicationfailure,butaredundancymanagershould
beabletodetectastoppageofdataflowinanamountoftimeveryclosetothetruedatarate
fromthephysicalsystem.Theredundancymanagershouldbeabletoswitchfromonesource
totheotherbasedsolelyonanobservationthatdatahasnotarrivedfromtheprimary
connection,buthasarrivedfromthebackupsystem.

SomesystemsuseCOMtimeoutsforlinkmonitoring.Thismaybeacceptableforcircumstances
whererelativelylongdataoutagesaretolerable,butwedonotrecommendrelyingonCOM
timeoutsforhotorwarmstandby.

SmartSwitchover

Thebehavioroftheredundancysystemduringaswitchovercanbesignificant.Forexample,
supposetheprimaryandsecondaryconnectionshavebothfailedforsomereason.Atypical
redundancymanagerwillbeginacycleofattemptingtoattachtooneandthentheotherOPC
Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com

Cogent Real-Time Systems Inc

serveruntiloneofthemresponds.Theredundancymanagerwillflipflopbetweenthetwo
indefinitely,injectingsleepperiodsbetweeneachflipfloptoreducesystemresourceload.This
sleepperiodisitselfasourceoflatency.Asmarterswitchovermodelistomaintainasource
healthstatusthatallowstheredundancymanagertoonlyswitchoverwhenasourcestatus
changes.Thisallowstheredundancymanagertoeffectivelyidle,orperformsimultaneous
reconnectionattempts,untilasourcestatuschanges,thenimmediatelyrespondwithout
introducingextralatency.Smarterswitchinglogiccanresultinsubstantiallyreducedsystem
loadandswitchovertimes.

ForcedSwitchingvsPreferredSource

Itisusefultobeabletoselectonedatasourceoveranother,evenifthecurrentlyattached
sourceishealthy.Anaveredundancymanagerwillforcetheusertoswitch,evenifthe
backupsystemisnotavailable.Thiswillagainresultinaflipflopbehaviorastheredundancy
managerattemptstoswitchtotheunavailablebackupsource.Amuchbetterapproachisfor
theredundancymanagertounderstandtheconceptofapreferredsourcethatcanbechanged
atruntime.Ifthepreferredsourceisavailable,theredundancymanagerwillswitchtoit.Ifthe
userwantstoswitchfromonesourcetoanother,hesimplychangesthepreferredsource.If
thatsourceisavailable,theswitchwillbemade.Ifitisnot,theredundancymanagerwillmake
theswitchonlywhenitbecomesavailable.Thiseliminatestheflipflopbehaviorwhileatthe
sametimeeliminatingthedatalossassociatedwiththeminimumoftwoswitchcyclesthatthe
naveredundancymanagerwillimpose.

AccessingRawData

Agoodhotredundancysystemwillgivetheclientapplicationaccessnotjusttotheredundant
data,butalsototherawdatafrombothsources.Thisgivestheclientapplicationtheoptionof
presentingdiagnosticinformationaboutthesystemonthefarsideoftheredundancy
manager.Mostredundancymanagershidethisinformationsothataclientapplicationwould
havetomakeandmanagemultipleconnectionstoaccesstherawdata,ifitispossibleatall.

Otheroptionsandfeatures

Inadditiontotheabovecapabilities,agoodredundancymanagermayofferadditionalfeatures
foryourconvenience.Itmightprovidetheoptiontorefreshtheentiredatasetatswitchover.
Maybeitwillsendoutemailsorevenlaunchadditionalprogramsateachswitchover.Thiscan
beusefulfornotifyingkeypersonnelofthesystemstatus.Itmaylogdiagnosticstoprovide
valuableinformationaboutthereasonsformakingtheswitch.Someredundancymanagerscan
connecttomultipleservers,andcreatemultipleredundantconnections.Otherscanletyou
workwithsubsetsofthedata.Anotherdesirablefeatureistheabilitytoassigntheprimaryand
secondarydatasources,andtotriggerafallbackfromthesecondarytotheprimarydatasource
oncetheproblemthatcausedtheswitchoverhasbeenresolved.

Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com

Cogent Real-Time Systems Inc

Ascontrolsystemscontinuetogrowincomplexity,andaswerelymoreandmoreonthem,
MelFarnsworthssituationwillbecomemorecommon,andmorecostly.Ifdataconnectivityis
crucialtothesuccessofthecompany,itwouldbewisetoconsiderthepossibilityofinstallinga
redundantsystem,andtoweightheoptionscarefullywhenchoosingthekeycomponents.

Foundedin1995,CogentRealTimeSystemsprovidesversatileandreliablemiddleware
productstoenablerealtimedataintegrationandaccessforindustrial,embedded,andfinancial
systems.CustomersincludeSiemens,ABB,Honeywell,IBM,GE,Statoil,Goodyear,BASF,
CadburyChocolate,andtheBankofCanada.Formoreinformation,pleasecontactCogentat
info@cogent.caorvisitourwebsiteatwww.cogentdatahub.com.Youcanalsocallusat+1
(905)7027851.

Copyright 2011, Cogent Real-Time Systems Inc.

www.cogentdatahub.com