WHITEPAPER

VIRTUALIZATION DIGITAL SUBSTATIONS

www.welotec.com
 Engineering Data Human Machine
workstation historians Interface

Windows Windows Windows

VIRTUALIZATION IN DIGITAL SUBSTATIONS Virtual

machine
Virtual
machine
Virtual
machine
ENSURING REDUNDANCY, STANDARDIZATION OF HARDWARE AND
Substation SCADA System
REDUCTION OF MAINTENANCE COSTS USING VIRTUALIZATION
Hypervisor (Hyper-V)
Gateway

IEC 61850 compliant hardware

Increasing computing requirements in substations
Workstation, historian and machine
In modern substations, an increasing number of field devices and automation components
interface, redundant and a higher degree
hardware
of automation means that more and more computers are required. Communication components and operator
terminals (HMIs) as well as SCADA systems require hardware that meets stringent requirements to ensure safe
operation of the substation environment. In many substations, there are separate computers for each individual
function, such as small embedded computers but also 19“ station computers.
Substation LAN
Through virtualization, many systems can be accommodated on one platform. Also, the construction of redun-
dant systems becomes easier. Virtualization originally comes from the area of large IT data centers and abstracts
Engineering IT Authentication
resources by inserting an additional
Surveillance & layer between application and hardware. This makes it possible to emulate
server and virtually
server provide services,
Access operating systems on a common hardware. For the user, the virtual object behaves
control
like a dedicated hardware or software object.
Windows Windows Linux

Virtual DevelopmentVirtual timeline of transformer station equipment

Virtual
machine machine machine Engineering, authentication
and surveillance
In substations, there are typically the following server,
applications that have a specific need for computing power and
Hypervisor (Hyper-V) redundant hardware
different requirements for operating systems or hardware resources:
IEC 61850 Network
IEC 61850•compliant
Intelligenthardware
operator terminals (HMIs) IEDs
• Authentication server with Lightweight Directory Access Protocol (LDAP) andMeasurement
RADIUS protocol
and merging units
• Workstations for maintenance technicians or operating personnel
• Historical data databases for analysis and monitoring
• Communication gateways and data concentrators
• SCADA systems for monitoring and control
• Other automation equipment, PQ management etc.

Substation Engineering Human Machine SCADA Data historians

Gateway Workstation Interface system and data bases

Substation LAN

Engineering Authentication Access Control Surveillance

server server server system server

IEC 61850 Network

IEDs
Measurement and merging units

Figure 1: Conventional computing structure of a transformer station

www.welotec.com
 App1 App1 App1 App1 App1 App1

VIRTUALIZATION IN DIGITAL SUBSTATIONS

App1 App1 App1 App1 App1 App1 OS OS OS

ENSURING
OS REDUNDANCY,
OS OS STANDARDIZATION
Virtual
machine
Virtual
machine OF HARDWARE AND
Virtual
machine
REDUCTION
Virtual OF MAINTENANCE
Virtual Virtual COSTS USING VIRTUALIZATION
Hypervisor (Hyper-V)
machine machine machine
These applications are operated on individual computers in many substations. However, this entails high main-
Hypervisor (Hyper-V) Host Operating System
tenance costs and is also not recommended for reasons of system redundancy. Each application uses hardware
tailored to its needs, which however represents a single point of failure. In order to expand each application
redundantly, a huge number of small computers must be installed, all of which would have to be kept in stock
Hardware Hardware
andCPU
maintained for possible replacement.
Memory CPU Memory

By using virtualization, applications on the same hardware can be operated completely separately. Different
Storage Periphals Storage Periphals
operating systems can be used for each application and the prioritization of the individual applications can be
assigned.

This makes it possible to provide several applications with a uniform, then redundantly operated hardware inf-
rastructure that is easy to maintain and adapt to specific conditions. Such an implementation, which is based on
the picture above, would look as follows:

Engineering Data Human Machine

workstation historians Interface

Windows Windows Windows

Virtual Virtual Virtual

machine machine machine
Substation Hypervisor (Hyper-V) SCADA System
Gateway

IEC 61850 compliant hardware

Workstation, historian and machine

interface, redundant hardware

Substation LAN

Engineering Authentication Surveillance &

server server Access control

Windows Windows Linux

Virtual Virtual Virtual

machine machine machine Engineering, authentication
and surveillance server,
Hypervisor (Hyper-V) redundant hardware
IEC 61850 Network
IEC 61850 compliant hardware IEDs
Measurement and merging units

Firgure 2: Virtualized substation

Substation Engineering Human Machine SCADA Data historians

www.welotec.com Gateway Workstation Interface system and data bases
VIRTUALIZATION IN DIGITAL SUBSTATIONS
ENSURING REDUNDANCY, STANDARDIZATION OF HARDWARE AND
REDUCTION OF MAINTENANCE COSTS USING VIRTUALIZATION
Benefits of using virtualized computing power

This simpler and fail-safe hardware infrastructure with virtualized application level creates a lot of added value
during planning, operation and the adaptation to new conditions:

• Reduction of investment costs through procurement of identical hardware

• Reduction of maintenance and repair costs
• Ensuring redundancy and reliability of individual components of the substation
• Unification of the IT and OT infrastructure
• Achievement of a security level that considers all compliance processes

Virtualization - A cost driver?

The investment and operating costs of redundant applications often appear to be high, since a large number of
computers have to be procured and operated in existing systems. With high-performance hardware, however,
different applications can be run on one and the same hardware platform (Fig. 2). The standardization of hard-
ware, IT and OT technologies results in economies of scale that make the use of virtualization very interesting.

Investment protection through virtual machines

Transformer substations generally use specialized software, which represents a significant portion of the total
cost of the IT environment. The original manufacturer may have disappeared from the market and no longer
provide new versions, or the existing license may not cover newer versions and runs only on legacy operating
systems such as Windows NT, 98 or XP, for which no current hardware is available. This legacy hardware can be
made available virtually, thus allowing the continued use of existing software.

Virtualizations in Transformer Stations - Design Considerations

Hardware requirements
The selection and dimensioning of the right hardware platform should play a major role in virtualization, as all
redundancy and cost considerations are based on this:

• A high-performance platform with sufficient computing capacity, multiple processor cores and suitable
memory equipment is required.

• Minimum 32 GB to 64 GB ECC-RAM (error-correcting code). ECC-RAM can correct data corruption in memory
areas to increase reliability in substation environments.

• Flash-based, maintenance-minimized drive systems with redundant design. Single-level cell-based (SLC-
based) memory is characterized by a high number of available write cycles, high write speeds and long
memory maintenance, but is also more expensive than other flash memory media such as iMLC (industrial
multilevel cell). However, SLC-based drive technology is the preferred choice for some applications such
as fast storage of large amounts of data for a long period of time. For disaster prevention purposes, the
drives should at least be operated in a RAID 1 network. iSLC storage media represent a cost-effective middle
course, offering many advantages of SLC media and placed between MLC and SLC regarding write cycles.

• The CPU, graphics unit, and I/O layer should natively support virtualization to enable multiple operating
systems to access this shared infrastructure optimally. Typical examples of terms used to support virtualiza-
tion technologies of a CPU include Intel VT-x. Substation computers that support this standard can be better
used for virtualization purposes. Processors with VT-d also enable guest systems to provide dedicated access
to hardware components such as LAN ports.

www.welotec.com
VIRTUALIZATION IN DIGITAL SUBSTATIONS
ENSURING REDUNDANCY, STANDARDIZATION OF HARDWARE AND
REDUCTION OF MAINTENANCE COSTS USING VIRTUALIZATION
• The computing unit must be able to operate safely under all environmental conditions. Therefore, extreme
temperature ranges from -30°C to +70°C, high electromagnetic compatibility as well as shock and vibrati-
on resistance are essential in this hardware. These requirements are described in the standards IEEE 1613,
IEC 61850-3, IEC 61000-4 and IEC 60255.

• The computer platform requires NICs capable of performing VLAN tags and trunking. It must also be able
to be equipped with several independent Ethernet ports. This supports the separation of different systems
and services per port.

• The network protocols HSR and PRP are required for new IEC 61850 applications. Station computers should
support HSR and PRP to be seamlessly integrated and used on all station levels.

• The hardware must be designed to be maintenance friendly and modular. For example, the replacement
of a defective power supply unit during operation via two redundant power supplies is required in order to
maintain operation during maintenance.

• The long-term availability of components of the computing unit must be ensured for a long period of time.
It is therefore important to select components that are as up-to-date as possible.

Hypervisor - But which one?

In many projects the virtualization solution Hyper-V played an important role. Hypervisors are basically divided
into two groups depending on the needs of the respective applications: Type 1, where the hypervisor is placed
directly on the hardware (so-called bare metal, see figure 3 on the left side), and Type 2, where a basic operating
system lies between the hardware and the hypervisor (see figure 3 right side). The selection of the appropriate
virtualization components should be carefully considered in the planning phase.

App1 App1 App1 App1 App1 App1

App1 App1 App1 App1 App1 App1 OS OS OS

OS OS OS Virtual Virtual Virtual

machine machine machine
Virtual Virtual Virtual Hypervisor (Hyper-V)
machine machine machine

Hypervisor (Hyper-V) Host Operating System

Hardware Hardware
CPU Memory CPU Memory

Storage Periphals Storage Periphals

Figure 3: Type 1 vs. Type 2 Hypervisor

Engineering Data Human Machine

workstation historians Interface
www.welotec.com
Windows Windows Windows

Virtual Virtual Virtual

machine machine machine
VIRTUALIZATION IN DIGITAL SUBSTATIONS
ENSURING REDUNDANCY, STANDARDIZATION OF HARDWARE AND
REDUCTION OF MAINTENANCE COSTS USING VIRTUALIZATION
Storage and resource management

It is necessary to evaluate which applications require which memory and resources. It must also be checked whe-
ther the hardware used supports the hypervisor and hardware drivers. Network functions (including support for
multiple VLANs and virtual switches as well as redundancy technologies such as HSR/PRP) should also be native
to the hardware. Furthermore, the hardware used should meet the requirements of IEC 61850-3.

Redundancy architecture

In the transmission and distribution networks, are very critical, it is common to use a high-availability redundan-
cy architecture. This includes the appropriate design of intelligent electronic devices (IEDs), gateways, SCADA
systems, network technology, cabling and much more. This design is made possible using central, redundant
systems for different applications, i.e. by virtualization, in a simple and cost-effective manner. However, extensive
planning steps should be carried out before implementation in order to guarantee optimum results and safe and
cost-effective operation. With a good redundancy architecture, network operators can protect themselves against
unplanned hardware failures and reduce single points of failure.

VLAN architecture and cost reduction through virtual Switches

Network infrastructures in substations can become very complex. Some applications within such network sta-
tions are time-critical, other data flows have high bandwidth requirements. By using VLANS, data traffic can be
prioritized and network performance increased. The virtual machines are not directly connected to a physical
Ethernet interface, but via a so-called internal private virtual switch (vSwitch). This then connects the individual
applications with the physical Ethernet port and serves as a bridge between the logical Ethernet ports of the VM
and the physical interfaces. It is possible to configure multiple VLANs on a vSwitch and send GOOSE traffic with
QoS requests over it.
Virtual switches reduce hardware costs by not requiring many physical Ethernet interfaces. Especially in HSR/PRP
networks, which are built redundantly, many cost-intensive HSR/PRP ports are eliminated.

IT/OT Convergence

The IT departments of many network operators are responsible for the selection of hardware and software, patch
management and the operation of communication networks. IT and OT technologies are moving closer using
virtualization, resulting in synergy effects. Even though there are different requirements for IT and OT systems,
numerous common elements can also be identified:
• Development of common standards and guidelines
• Use of the common infrastructure for maintenance and service
• Use of identical or similar technology platforms
• Unification of license agreements for software
• Development of common governance
• Managing IP address management for WAN integration

www.welotec.com
VIRTUALIZATION IN DIGITAL SUBSTATIONS
ENSURING REDUNDANCY, STANDARDIZATION OF HARDWARE AND
REDUCTION OF MAINTENANCE COSTS USING VIRTUALIZATION
Security challenges and requirements

The North American Electric Reliability Corporation (NERC) has developed a series of standards that describe the
protection of critical infrastructures against cyber threats. Parallel to this, there is the NIS Directive in Europe. The
NERC standards are mandatory for companies in North America. For companies in Europe it is the NIS Directive. In
addition, most utilities around the world have also committed themselves to these standards to varying degrees,
as they represent an industry-recognized best practice approach to most cyber security issues.
Cyber security design should include the following key components:

• Defense-in-depth concepts with firewall layer and demilitarized zone (DMZ)

• Role-based access management
• Encryption of remote access connections
• Secure authentication and authorization
• Protection with antivirus and anti-malware software
• Consistent patch management for all software and firmware components
• Attack detection through intrusion detection systems

Summary

The ever-increasing digitalization of the world will not stop at the power grids either. E-mobility and renewable
energies will increasingly drive the grid and thus also the demands on computing power for control. It is therefo-
re important to make the right decisions when selecting computing elements in order to enable digitization and
secure operation of substations.
IT and OT projects in the energy industry are always complex and require extensive and careful planning. There-
fore, close, reliable cooperation in product selection and adaptation is necessary.

A Whitepaper by Welotec GmbH, Authors:

Jos Zenner Thomas Schulte Dennis Kock

Chief Technology Officer (CTO) Product Manager Industrial & Edge Computing Head of Product Management
j.zenner@welotec.com t.schulte@welotec.com d.kock@welotec.com

www.welotec.com
Welotec GmbH
www.welotec.com

Zum Hagenbach 7
V 1.0 October 2019

D-48366 Laer
Fon: +49 (0)2554/9130-00
Fax: +49 (0)2554/9130-10
info@welotec.com