Considerations,

Best Practices
and Requirements
for a Virtualised
Mobile Network
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

About the GSMA

The GSMA represents the interests of mobile operators
worldwide, uniting nearly 800 operators with almost 300
companies in the broader mobile ecosystem, including handset
and device makers, software companies, equipment providers
and internet companies, as well as organisations in adjacent
industry sectors. The GSMA also produces industry-leading
events such as Mobile World Congress, Mobile World Congress
Shanghai, Mobile World Congress Americas and the Mobile 360
Series of conferences.
For more information, please visit the GSMA corporate website
at www.gsma.com. Follow the GSMA on Twitter: @GSMA.
Network 2020
The GSMA’s Future Networks Programme is designed to
help operators and the wider mobile industry to deliver all-IP
Contents
networks so that everyone benefits regardless of where their
starting point might be on the journey.
The programme has three key work-streams focused on:
The development and deployment of IP services, The
evolution of the 4G networks in widespread use today, 1 Introduction 3 5.1.4 NFV carrier-grade reliability scope 24
The 5G Journey developing the next generation of mobile 1.1 Scope 4 5.2 Carrier Grade Service – Requirements 26
technologies and service. for NFV Reliability
1.2 Definitions 4
For more information, please visit the Network 2020 website 1.3 Abbreviations 5 5.3 NFV Reliability Assessment Standard 26
at: www.gsma.com/network2020 5.4 Best Practices 28
1.4 References 6
5.4.1 Use case scenarios 28
2 NFV reference architecture 7 5.5 Conclusion 29

3 Requirements for Network Orchestrators 9 6 Migration 31

3.1 Introduction 10 6.1 Introduction 32
3.1.1 Network Services 10 6.1.1 Migration from physical network to 32
3.1.2 Virtualised Network Functions 11 virtualised network
3.1.3 Network Functions Virtualisation 11 6.1.2 Software upgrades 32
With thanks to contributors: Infrastructure 6.1.3 Interoperability 32
AT&T
3.2 NFV Orchestrator functional requirements 11 6.2 Best practices 32
China Mobile
China Telecom 3.3 Other NFV related architectures 12 6.2.1 Migration 32
China Unicom 3.4 Best Practices 14 6.2.2 Software upgrades 33
KDDI
3.4.1 Planning on NFV based network 14 6.3 Requirements 34
KT
LG Uplus management and orchestration 6.3.1 Migration 34
NTT DoCoMo architecture
6.3.2 Software upgrades 34
Orange 3.4.2 Example of 5G Telco-MANO architecture 15
Proximus
America Movil 7 Performance benchmark for NFV 35
4 Virtualised network security 17 infrastructure
SK Telecom
Telecom Italia 4.1 Introduction 18 7.1 Performance benchmark scope 36
Telefónica 4.1.1 Maturity 18
Telekom Austria 7.2 State of the art in NFV performance 37
Telenor Group 4.2 Architectural Challenges 18 benchmarking
TeliaSonera 4.3 Best practices 19
T-Mobile US 8 Vertical interoperability 41
4.4 Conclusions 19
United States Cellular
4.5 Actions 19 8.1 Introduction 42
Vodafone
Ascom Network Testing 8.2 Requirements for vertical interoperability 42
Cisco 5 Carrier grade NFV/ Reliability 21 8.3 Challenges and potential risks 42
Ericsson
5.1 Introduction 22 8.4 Best Practices 43
Gemalto
HP 5.1.1 Carrier-grade 22
Huawei 5.1.2 The essential value of carrier-grade 22
Nokia
reliability
Oracle America
Samsung Electronics 5.1.3 NFV carrier-grade reliability challenges 23
ZTE
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

1
Introduction

2 3
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
1.1 Scope 1.2 Those are Definitions
This document outlines the key considerations in
Term Description
the deployment of network virtualisation in a mobile
Lifecycle management Set of functions required to
network environment. The topics covered within manage the instantiation,
maintenance and termination of a
represent solutions to the potential obstacles mobile
Virtualised Network Function or
operators may face when wishing to capitalize on
Network Service
network virtualisation (covering both Network Functions Network controller Functional block that centralizes
Virtualisation and Software-Defined Networking). some or all of the control and
management functionality of a
This document provides an overview of the steps network domain and may provide
an abstract view of its domain to
mobile operators should take to adopt this technology other functional blocks via well-
and where appropriate, provide an indication of defined interfaces
what they will need to complete the work and which
Network Function Functional block within a
external organisations are best placed to deliver it. network infrastructure that has
well-defined external interfaces
The document also outlines a number of examples and well-defined functional
behaviour
and approaches that have been taken by operators
Network Functions Principle of separating Network
to identify and address the gaps. These best practice Virtualisation Functions from the hardware
examples are provided as guidance and do not they run on by using virtual
represent the consensus of the GSMA members hardware abstraction
participating to this activity. Network Functions Functional block that manages
Virtualisation Orchestrator the Network Service lifecycle
and coordinates the management
of Network Service lifecycle,
Virtualised Network Function
lifecycle and NFV infrastructure
to ensure an optimized allocation
of the necessary resources and
connectivity
Network Functions Totality of all hardware and
Virtualisation Infrastructure software components that build
up the environment in which
Virtualised Network Functions
are deployed
Network service orchestration Subset of Network Functions
Virtualisation Orchestrator
functions that are responsible
for Network Service lifecycle
management
Network service descriptor Template that describes the
deployment of a Network Service
including service topology
as well as Network Service
characteristics such as Service
Layer Agreements for the
Network Service on-boarding
and lifecycle management of its
instances
Network Service Composition of Network
Functions and defined by its
functional and behavioural
specification
Orchestration Type of composition where one
particular element is used by
the composition to oversee and
direct the other elements
Quota Upper limit on specific types of
resources
4 5
CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
Resource orchestration Subset of Network Functions ETSI European Telecommunications
Virtualisation Orchestrator Standards Institute
functions that are responsible
ETSI GS ETSI Group Specification
for global resource management
governance FCAPS Fault, Configuration, Accounting,
Performance and Security
Service lifecycle Set of phases for realizing a
service from conception and GS-O Global Service Orchestration
identification to instantiation and
retirement
ICT Information and Communications
Technology
Software-Defined Networking A set of techniques that enables
IoT Internet of Things
to directly program, orchestrate,
control and manage network IPC Instructions Per Cycle
resources, which facilitates the
iTLB/dTLB instruction Translation Lookaside
design, delivery and operation
Buffer / data Translation
of network services in a dynamic
Lookaside Buffer
and scalable manner
ISB Industry Standard Benchmarking
SDN Orchestration A process that oversees and
directs a set of software-defined ISG Industry Specification Group
networking activities and
KPI Key Performance Indicator
interactions with the objective of
carrying out certain work in an KVM Kernel-Based Virtual Machine
automated manner.
L1/L2/LLC Level 1/Level 2/ Last Level Cache
Virtualised Infrastructure Functional block that is
Open-O Open Orchestrator Project
Manager responsible for controlling and
managing the Network Functions OPEX Operating Expense
Virtualisation Infrastructure
OPNFV Open Platform for NFV
compute, storage and network
resources, usually within one OS Operating System
operator’s Infrastructure Domain OSM Open Source MANO project
Virtualised Network Function Implementation of an Network OSS Operations Support Systems
Function that can be deployed on
OVS Open vSwitch
a Network Function Virtualisation
Infrastructure PGW Packet Data Network Gateway
Virtualised Network Function Internal component of a PNF Physical Network Function
Component Virtualised Network Function
RAN Radio Access Node
providing a defined sub-set
of that Virtualised Network RFC Request For Comments
Function’s functionality.
MANO Management and Orchestration
MME Mobility Management Entity
1.3 Abbreviations
MVNO Mobile Virtual Network Operator
Term Description
NextGen Next Generation
API Application Programming
NF Network Function
Interface
NFV Network Functions Virtualisation
ATIS The Alliance for
Telecommunication Industry NFV-O Network Functions Virtualisation
Solutions Orchestrator
BSS Business Support Systems NFVI Network Functions Virtualisation
Infrastructure
CAPEX Capital Expenditure
NOC Network Operations Centre
CNI Critical Network Infrastructure
NS Network Service
CPU Central Processing Unit
SDN Software Defined Networking
DPDK Data Plane Development Kit
SDN-O Software Defined Networking
DUT Device Under Test
Orchestration
ECOMP Enhanced Control, Orchestration,
SHV Standard High Volume
Management & Policy
SGW Serving Gateway
EOSL End of Service Life
UMO Unified Management and
EPC Evolved Packet Core
Orchestration
EM Element Management
vACL virtual Local Area Network
Access Lists
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

vCG-NAPT virtual Concatenation Group [17] GS ETSI NFV ETSI GS NFV 001: “Network Function

2
Network Address Port 001 Virtualisation (NFV); Use Cases”
Translation
[18] GS ETSI TST ETSI GS NFV-TST 001: “pre-
vEPC virtual Evolved Packet Core 001 deployment testing, report on
validation of NFV environments and
vFW virtual Fire Wall
services”
vMME virtualised Mobility Management
[19] GS ETSI TST ETSI GS NFV-TST 002: “Testing
Entity
002 Methodology; Report on NFV
vPE virtual Provider Edge Interoperability Testing Methodology”
vSwitch virtual Switch [20] GS ETSI PER ETSI GS NFV-PER 001: “NFV
001 performance and portability best
VIM Virtualised Infrastructure
practices”
Manager

NFV reference
[21] GS ETSI IFA ETSI GS NFV-IFA 003 V2.1.1: “Network
VM Virtual Machine
003 Functions Virtualisation (NFV);
VNF Virtualised Network Function Acceleration Technologies; vSwitch
benchmarking and acceleration
VNFC Virtualised Network Function
specification”
Component

architecture
[22] draft-ietf- https://datatracker.ietf.org/doc/draft-
bmwg-virtual- ietf-bmwg-virtual-net/
1.4 References net-04

Ref Doc Number Title [23] draft-ietf- IETF: draft-ietf-bmwg-ipsec-term-12.

bmwg-ipsec- txt, “Terminology for Benchmarking
[1] GS ETSI NFV Network Functions Virtualization term-12 IPsec Devices”.
002 (NFV); Architectural Framework
[24] draft-ietf- IETF: draft-ietf-bmwg-ipsec-meth-05.
[2] GS ETSI NFV- Network Functions Virtualisation bmwg-ipsec- txt, “Methodology for Benchmarking
MAN 001 (NFV); Management and Orchestration meth-05 IPsec Devices”.
[3] GS ETSI NFV- Network Functions Virtualisation [25] draft-ietf- IETF: draft-vsperf-bmwg-vswitch-
IFA 009 (NFV); Management and bmwg- opnfv-01.txt, “Benchmarking Virtual
Orchestration; Report on Architectural vswitch- Switches in OPNFV”.
Options opnfv-01
[4] GS ETSI NFV- Network Functions Virtualisation [26] draft-kim- IETF: draft-kim-bmwg-ha-
IFA 010 (NFV); Management and bmwg-ha- nfvi-01.txt, “Considerations for
Orchestration; Functional nfvi-01 Benchmarking High Availability of NFV
requirements specification Infrastructure”.
[5] ITU-T Y.3300 Framework of software-defined [27] GS ETSI NFV- ETSI GS NFV-REL001 “Network
networking REL001 Functions Virtualisation (NFV);
[6] ISO/IEC Reference Architecture for Service Resiliency Requirements”
18384-1 Oriented Architecture (SOA RA) Part 1: [28] ETSI NFV-SEC ETSI GS NFV-SEC 007 “Network
Terminology and Concepts for SOA 007 Functions Virtualisation (NFV); Trust;
[7] NGMN 5G white paper Report on Attestation Technologies
and Practices for Secure Deployments”
[8] AT&T Ecomp Enhanced Control, Orchestration,
Management & Policy Architecture; [29] ETSI GS NFV- ETSI GS NFV-SEC 013 “Network
http://about.att.com/content/dam/ SEC 013 Functions Virtualisation (NFV) Release
snrdocs/ecomp.pdf 3; Security; Security Management and
Monitoring specification”
[9] Open-O Open-O architecture; www.open-o.org
[30] ETSI GS NFV- ETSI GS NFV-SEC 012 “Network
[10] OSM Open Source MANO; http://osm.etsis. SEC 012 Functions Virtualisation (NFV) Release
org/ 3; Security; System architecture
[11] ETSI GS NFV- Network Function Virtualisation (NFV); specification for execution of sensitive
REL006 V0.0.3 Reliability; Specification on Software NFV components”
(2016-07) Update Process [31] ETSI TS 103 ETSI TS 103 308 “CYBER; Security
[12] Yardstick https://wiki.opnfv.org/display/ 308 baseline regarding LI and RD for NFV
yardstick and related platforms”

[13] Bottlenecks https://wiki.opnfv.org/display/

bottlenecks
[14] StorPerf https://wiki.opnfv.org/display/storperf
[15] VSPerf https://wiki.opnfv.org/display/vsperf/
VSperf+Home
[16] CPerf https://wiki.opnfv.org/display/cperf

6 7
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

Figure 1 shows the NFV reference architectural and The NFV Orchestrator is responsible for the
functional blocks [1]. The functional blocks are: management and orchestration of NFV infrastructure,

• Operations Support Systems (OSS) and Business
Support Systems (BSS).
• Element Management (EM);
• Virtualised Network Function (VNF);
• Service, VNF and Infrastructure Description;
• VNF Manager(s);
• NFV Orchestrator;
• Virtualised Infrastructure Manager(s) (VIM);
• NFV Infrastructure (NFVI), including hardware and
virtual compute, storage and network resources
A VNF is an implementation of a Network Function
that can be deployed in an NFV Infrastructure.
Examples of mobile network functions are Mobility
Management Entity (MME), Serving Gateway (SGW),
Radio Access Node (RAN) and Packet Data Network
Gateway (PGW).
3
as well as the software resources and delivering the
network services on the NFV infrastructure. The ISG
ETSI NFV [3] identifies the following management and
orchestration data repositories:
• VNF Catalogue;
• NS Catalogue;
• NFV Instances repository;
• NFVI Resources repository
The NFV Orchestrator has two main responsibilities:
• Network Service Orchestration: managing the
lifecycle of network services. This includes the
Requirements for
Network Orchestrator
management of the Network Services templates and
VNF Packages. The Network Service Orchestration
provides the management of the instantiation of
VNFs, in coordination with VNF Managers;
• Resource Orchestration: providing an overall view of
the resources to which it provides access and hides
the interfaces of the VIMs present below it

Figure 1: NFV architecture

Lifecycle management
of Network Services

Management & Orchestration

NFV
OSS/BSS Orchestrator Service,
VNF and
Infrastructure
Description
EM EM EM EM
NFV
Manager(s)
VNF VNF VNF VNF VNF

Virtualised
NVF Infrastructure (NVFI) Infrastructure
Manager(s)

Management of the services Control management of Lifecycle management

provided by the NF the virtualised resources of VNF instances

Source: GS ETSI NFV-002 Network Function Virtualization (NFV); Architecture Framework

8 9
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

3.1 Introduction 3.1.1 Network Services 3.1.2 Virtualised Network Functions 3.2 NFV Orchestrator functional requirements
Network Functions Virtualisation requires a new set of The Network Service Orchestration is responsible for The management and orchestration elements of a VNF The Network Orchestrator shall fulfil all requirements
management and orchestration functions to be added the Network Service lifecycle management including include fulfilment, assurance and security management, specified in clause 6 of ETSI NFV-IFA 010 [5]
to the current model of operations, administration, operations such as: however, the focus in NFV is the decoupling of the VNF applicable to an NFVO. These requirements include
maintenance and provisioning. The Network Functions • On-board and management of Network Service software from the hardware. The decoupling of Network functional requirements for VNF lifecycle, Network
Virtualisation Management and Orchestration has the Descriptors; Functions from the physical infrastructure results in a Service lifecycle and virtual resource management
role of managing the NFVI and control the allocation of new set of management functions that are focused on summarized as follows:
• Instantiate Network Service;
resources needed by the NSs and VNFs. the creation and lifecycle management of virtualised
• Scale Network Service; • Network Service:
resources for the VNF, referred to as VNF Management.
The management and orchestration takes place at • Update Network Service; VNF Management functions are responsible for the VNF’s • lifecycle management (instantiation, scaling,
three different levels as depicted in Figure 2. lifecycle management including operations such as: updating, termination);
• Terminate Network Service
• information management (NS descriptor);
• Instantiate VNF (create a VNF instance using the
• performance management;
VNF on-boarding artefacts);
• fault management
• Scale VNF (increase or reduce the capacity of the
Figure 2: Management & Orchestration at different abstraction levels VNF); VNF:
• Update and/or Upgrade VNF (support VNF • lifecycle management (instantiation, scaling,
software and/or configuration changes of various termination);
complexity);
• information (VNF package) management;
NS • Terminate VNF (release VNF-associated NFVI
VNF • configuration management (initial configuration and
resources and return it to NFVI resource pool)
NVF Orchestrator (NFVO) update);
VNF 3.1.3 Network Functions Virtualisation Infrastructure • performance management;
VNF Network Functions Virtualisation Infrastructure • Indicator management
(NFVI) resources under consideration are both
• fault management
virtualised and non-virtualised resources, supporting
virtualised network functions and partially virtualised Virtualised resource management:
VNF network functions.
• direct/indirect, reservation;
VNFC
Virtualised resources are offered for use through • capacity, fault, performance;
VNFC VNF Manager (VNFM) abstracted services, for example: • network forwarding path;
• Network, including: networks, subnets, ports, • information;
VNFC addresses, links and forwarding rules, for
• quota, allowance
the purpose of ensuring intra- and inter-VNF
connectivity; Other requirements are also considered for:
• Compute including machines, and virtual machines, • Multi-Tenancy (Tenant management);
as resources that comprise both CPU and memory; • Infrastructure resource management;
• Storage, including: volumes of storage at either • Software image management;
Virtualised Infrastructure
block or file-system level
Manager (VIM) • NFV acceleration management;
• Security consideration;
Virtual Switches SDN
Storage
machines & Routers controllers • Policy administration

Source: Orange

10 11
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

3.3 Other NFV related architectures
In NFV reference architecture, the VIM can play
the role of an SDN application, sitting on top of the
northbound interfaces of the SDN controller. The
VIM delegates to the SDN controller the connectivity
management of virtualised network resources that
are required by network services and their constituent
network functions. This architecture does not propose
a direct interface between the NFV-O and SDN
controllers. However, in some architecture (e.g. AT&T
ECOMP architecture), a Master Orchestrator for NFV
and SDN acts as an NFV-O and as a direct consumer
of the APIs exposed by an SDN controller.
In the case of the OSM project, as shown in Figure
4, the SDN controller is a well-defined part of the Figure 4: OSM architecture mapping onto ETSI NFV framework elements
resources managed by the Resource Orchestrator,
rather than accessing it exclusively through the VIM
interface. This provides finer control of network
configuration, decouples resource and cloud-native GUI & Design-Time Tools
lifecycle management and supports multi-VIM services OSS/BSS
at the Resource Orchestrator level.
Network Service Orchestrator
EMSs
Specific
NFV Management and Orchestration
VNFMs

VNFs
Figure 3: ECOMP orchestration architecture
Resource Orchestrator
Ve-Vnfm VNF Configuration
PNFs (Includes VIM/SDN
& Abstraction
Management & Orchestration Connectors)

OSS/BSS
NFV Orchestrator NFVI

Application Nf-Vi
Controller
ODL
EM EM EM EM
OpenVIM OpenStack OpenVIM

VNF VNF VNF VNF VNF Network Floodlight

Controller
Main NFV reference points

Infrastructure Source: OSM Project

Cloud Controller
NVF Infrastructure (NVFI)

In another architectural approach, the SDN

Orchestrator can be separated from the NFV
Source: ECOMP www.openecomp.org Orchestrator (e.g. Open-O architecture) to manage
the SDN service connectivity independently from NFV
(e.g. for VPN on Demand use case). In this approach, a
Global Service Orchestrator GS-O is placed on the top
of the SDN-O and NFV-O orchestrators.

12 13
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

Figure 5: Open-O orchestration architecture Figure 6: possible NFV based network management and
orchestration architecture
GS-O
GUI Portal
UMO (Unified Management
Abstract NBI Legacy OSS
and Orchestration)
Service Service Service
Lifecycle Mgr. Decomposer Formation

EMS (Legacy) EMS (New) VNFM

SDN-O NFV-O
GUI Portal GUI Portal

Abstract NBI Abstract NBI

TAS CSCF SBC vTAS vCSCF vSBC
SDN Lifecycle Mgr.
SDN Monitor NVF Monitor PNFs VNFs
VPN
NS Lifecycle Mgr.
SDN Res. Mgr. Traffic Optimise NVF Res. Mgr.
VIM
Source: China Mobile
Abstract SBI Abstract SBI

SDN ACCESS/WAN SDN EMS/NMS VIM
Driver Controller Drivers Driver Drivers
Source: Open-O www.open-o.org
3.4 Best Practices
3.4.1 Planning an NFV based network management
and orchestration architecture
Figure 6 shows an example of an architecture for a
future NFV based network management strucuture
from an implementation based on China Mobile’s
network. In this architecture, the complete network
management system can be divided into two parts.
The right section is a sub-system managing VNFs
under UMO (Unified Management and Orchestration).
The UMO is China Mobile’s NFV based network
management and orchestration system. The left
section shows the legacy OSS: in order to avoid having
to reconstruct the existing sub-system, the legacy
OSS will stay unchanged and will continue to manage
existing PNFs. An interface between legacy OSS and
UMO is not precluded and could be a proprietary
interface, based on operators’ requirements.
SDN NVF SDN VNFM VIM
Driver Controller Drivers Driver Drivers The UMO is a combination system that has the
capability of network orchestration, VNF and NS
lifecycle management. It also covers policy design
and management, resource management and VNF
FCAPS management.
The benefit of this architecture is that the role and
responsibilities of legacy OSS and UMO are clear,
separate and defined. The UMO can be considered
a unified system designed to manage the whole
virtualised network. All the information needed for
a mobile network service, VNF and NS resource
management will be handled by the UMO only, which
has no impact on the legacy network.
3.4.2 Example of 5G Telco-MANO architecture
ETSI’s Network Functions Virtualisation (NFV)
Industry Specification Group (ISG) is defining NFV-
MANO architecture. This is a framework for the
management and orchestration of all resources in the
cloud computing infrastructure as well as NS and VNF
lifecycle management.
Some network functions are hard to transform to
virtualised network functions (VNFs) and will remain
as physical network functions (PNFs).
These hybrid physical and virtual network
environments should be considered for a 5G Telco
management and orchestration architecture. 5G Telco-
MANO is required to provide end-to-end network
provisioning for 5G network slicing on demand and
one-view network monitoring instead of monitoring
physical and virtual network(s) separately. 5G Telco-
MANO should also be easy to implement.

14 15
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

Figure 7 shows an example of a 5G Telco-MANO In addition to NFV-MANO defined interfaces, Figure

architecture. 6 shows interfaces between an E2E orchestrator

In this architecture, NMS-Assurance is used for end-
to- end network assurance. NMS-Assurance is the
most widely used network assurance system in PNF
networks and can be extended as a virtual networks
assurance system with the help of EMS and VNF. Thanks
to the reuse of NMS- Assurance, end-to-end network
assurance can be easily implemented to avoid the
complexity of reconstructing a system from scratch.
4
and domain controllers as well as between domain
controllers and EMS that are necessary for the 5G
Telco MANO architecture to work.
By following this approach, a 5G Telco-MANO, end-
to-end on-demand network provisioning and one-
view network assurance in physical and virtual hybrid
networks can be achieved.

Virtualised
E2E orchestrator and domain controllers are new
elements designed to support end-to-to-end network
provisioning and configuration. In each network domain,
a network controller (for example, Transport-SDN,

network security
Cloud SDN) will provision and configure the network
function with an open interface regardless of whether
they are physical or virtualised network functions, with
the interfacing of an E2E orchestrator. In this way, an
E2E orchestrator can provision a 5G network slice using
both PNFs and VNFs end-to-end.

Figure 7: Possible end-to-end management and orchestration

architecture for 5G

Customer Portal/OSS/BSS

E2E Infra Manager

NMS E2E Orchestrator

SON
(E2E Assurance) (E2E Provisioning/Control

Repository NFV-O

Domain Controller
EMS VNFM
(Adaptation)

EMS

VNF
PNF (vCore-C, vCore-U..)
(AU, OLT...)
VFVI VIM

Source: KT

16 17
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
4.1 Introduction
NFV can greatly amplify existing security problems
in terms of impact. The vulnerabilities are similar
to those of today, but instead iNFV concentrates
them in one place and increases the likelihood of
a common mode failure. In many ways, it puts all
our “security eggs in one basket”. In traditional
telecommunications equipment, a number of factors
helped to frustrate would be attackers such as physical
security, proprietary software, hardware, installation
and configuration and reduced the ability to exploit
vulnerabilities.
4.1.1 Maturity
No vendor is at an advanced stage when it comes to
compliance with national security regulations regarding
NFV. A number of interested parties are working hard
with standardisation bodies to include and embed
security from the ground up. We do not anticipate
seeing mature NFV products/ solutions implementing
the latest security standards (e.g. ETSI NFV SEC)
in the near future. Therefore, any product currently
available today is highly unlikely to have the required
security built in. Indeed, many NFV vendors are new
to telecommunications security concepts. The cloud
technologies being used to underpin NFV are not of
the maturity required to underpin Critical Network
Infrastructure (CNI) and iits related obligations.
The main threat vectors for a virtual network remain
fundamentally unchanged:
• Loss of availability: Attacks that result in crashing
a virtual network element or rendering it unusable
through flooding/denial of service;
• Loss of confidentiality: Either caused by
eavesdropping or leakage of sensitive data;
• Loss of integrity: Resulting from a modification of
data during transit (man-in-the-middle-attack)
or in the virtual network element, as well as from
unauthorised access to a virtualised network function;
• Loss of control: Loss of control can take place at
the network level where the attacker controls the
network exploiting a protocol or implementation
flaw or at virtual function level
Another threat that needs to be considered in
virtualised networks is the possible absence of a single
entity overseeing the whole network. This is because
a virtual network (Figure 1) can be easily subdivided in
to different administrative domains each with different
elements requiring the establishment of secure
interaction between themselves.
18
4.2 Architectural Challenges
This section lists the security aspects that
operators deploying a virtualised network need
to take into account.
• Integrity of infrastructure and VNF
• Several Trust Domains:
–– A Trust Domain is a collection of entities that
share security policies
–– Even in the context of a single CSP:
–– a dedicated trust domain for LI in addition to
an admin trust domain
–– Regulation geographical constraints with location
attestation ensuring that LI can only take place
on known POI/IAP
• Separation of LI function executions from other VNF
and Hypervisor is considered a requirement
–– Segregation of Lawful Interception information/
flows. Monitoring behaviour of infrastructure
should not decrease the confidentiality of a
Lawful Interception solution
–– Encrypted targets DB in a separate/secured
context
–– Secure Encryption
–– a location to store “properly” keys ensuring that
they can’t be compromised by e.g. a privileged
hypervisor manager or other process.
–– Secure execution
CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
4.3 Best practices 4.4 Conclusions
The GSMA Fraud and Security Group’s (FASG) • Security for NFV is still a work in progress
preliminary discussions on NFV security highlighted 5 • Security involves all NFV architecture components
key security risks operators need to take into account: (VNF level, NFVI, MANO)
• Legal and Regulatory Compliance failure: Core risks
• LI community requires the strongest requirements
are about geographic deployment and security of
• LI Requirements / Security for LI Function will
solutions, particularly in heterogeneous environments;
depend on National Regulations / Countries
• Isolation Failure: Core risk is around functions
• Still quite difficult to establish the right level of
“escaping” from allocated resources, enabling
requirements for RFI/RFP
prejudicial data access to extensive areas of
information (at rest, in motion or in memory);
4.5 Actions
• Denial of Service: Risks include flooding of public
interface or resource exhaustion (e.g. on internal • All vendors must be able to credibly answer (with
network or memory); evidence) a set of basic questions around the
security of their NFV products in accordance with
• Topology Validation and Enforcement: Risk is
ETSI TS 103 308 [30]
around malicious configuration (e.g. VM layer
misconfiguration); • Operators should consider a delay in the use of
NFV where sensitive functions are involved, unless
• Security Logging and Incident Management:
appropriate mitigations are available
Risks arise from failures to log or manage issues
• Government regulatory advice should provide
from complex interactions across domains
greater clarity, now, on the potential security risks
FASG also elaborated on mitigation strategies for the around NFV, and be able to legally and competently
above risks highlighting the importance of a strong deal with this topic
cooperation between operators, manufacturer and
where appropriate regulators. Three recommendations
are made:
• Vendors shall be required to adopt best-of-breed
security features as specified by ETSI NFV SEC and
operators:
–– Location attestation means
–– Root of Trust (NFV SEC 007 [27]).
–– Multiple Trust Domains (NFV SEC 013 [28]).
–– Execution Enclaves for Sensitive Applications
(NFV SEC 012 [29])
• The open-source community needs to be motivated
to include basic security measures. A “fixing as we go
along” approach is not suitable for this environment;
• Everyone needs to develop a secure architecture
with separate zones of trust
19
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

5
Carrier grade
NFV/ Reliability

20 21
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

5.1 Introduction
5.1.1 Carrier-grade
In telecommunications, “carrier-grade” refers to a
system, or hardware or software component that is
extremely reliable, well tested and proven. Carrier
grade systems are tested and engineered to meet or
exceed the “five-9s” (99.999%) availability standards,
that provide resiliency and fast fault recovery.
Product or service development within the
telecommunications industry has traditionally followed
rigorous standards for stability, protocol adherence
and quality, reflected by the use of the term ‘carrier-
grade’ to designate equipment demonstrating this
reliability. Over time, telecom service providers have
engineered an extensive range of sophisticated
features into their networks, to the point where they
can guarantee their high reliability.
However, it is difficult to achieve the same grade of high
reliability in each component of the NFV network. What
is important for service providers is to be able to offer
carrier grade services that meet the required availability,
reliability and performance requirements for end-to-end
voice, video, data or converged-services.
Figure 8: The benefits of Carrier-grade NFV for Operators
5.1.2 The essential value of carrier-grade reliability
Telecom networks must be always-on and guarantee
a level of service because society, business and
industries increasingly depend on reliable connections
for both routine and critical communications.
Always-on reliability is mandatory and telecom service
providers have built their networks, reputations and
revenue streams on a foundation of carrier-grade
reliability. A carrier-grade network guarantees a ‘five-
9s’ availability standard, that allows for no more than
5.27 minutes of downtime per year per service. If there
is just one failure in the system, the NOC (Network
Operations Centre) will be notified and proceed with
remediation in less than 5 minutes so that the five-9s
target can still be met. Hence there is an overwhelming
need to automate the entire process, including quick
service recovery processes and to provide a seamless
transfer from the failing element to healthy elements.
This level of service is typically required by high-value
enterprise customers who often will pay a premium for
Service Level Agreements that specify high availability.
5.1.3 NFV carrier-grade reliability challenges
With all the industry initiatives around NFV, network
reliability has become a hot topic, as shown in a survey
published by Heavy Reading in Jan 2015, as well as
some challenges experienced by the industry when
implementing NFV:
• Service recovery in a multi-layer or multi-vendor
environment
• Carrier-grade reliability when NFV has components
with a lower grade of reliability
• New skills and process requirements
• Multi-component interoperability management
• Security concerns
• Complexity in E2E problem demarcation
Figure 9: Challenges faced by the NFV Testing Market
Carrier-grade scalability and robustness
Seamless control and provisioning of physical
and virtual networking infrastraucture
Openness and interoperability
Real-time and dynamic provisioning
NFV global reach and cross administration
Other
When service providers refine their plans to introduce
NFV into their networks, they are also reviewing
the implications for high reliability on a network
infrastructure that incorporates virtualised functions.
There are numerous initiatives underway currently
to specify, align and promote NFV carrier-grade
capabilities for achieving efficient carrier-grade
NFV solutions. These include ETSI NFV Proof of
Concept, Open-Platform for NFV project, Carrier
Network Virtualisation Awards, ATIS (The Alliance for
Telecommunication Industry Solutions) and various
supplier ecosystems.
27.2%
23.7%
17.1%
13.5%
10.3%
8.2%

0% 5% 10% 15% 20% 25% 30%

Reduce Improve
Cost of Poor Customer Source: Heavy Reading Mobile Networks Insider
Quality Churn

Benefits
Increase Protect
O&M Efficiency Revenue

Enhance
Brand
Image

22 23
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

5.1.4 NFV carrier-grade reliability scope 1. Each product should be highly reliable and fault
Practically, there are 3 key areas in achieving NFV-driven resilient. Different types of resiliency can be applied
carrier-grade reliability: product, network deployment such as “1+1” or “N+M” (simple 1+1 redundancy
(design and integration) and network maintenance. is deprecated as a recommended strategy
for resilience). For stateful applications, data
synchronization and restoration mechanisms should
be in place for state continuity in case of component
Figure 10: End-to-End Carrier-grade Reliability Scope failure. Furthermore, to respond to a failure of a
physical or virtual element within an NFV platform,
the management software must be able to detect
Network Deployment failed components, hosts, or virtualisation solutions
Product Network Maintenance
(Design & Integration) (e.g. VMs) immediately and recover automatically,
if possible through an autonomous response
Unified Monitoring
mechanism (example: virtual machine migration).
Application
3. Most of the challenges are in network maintenance.
Even if a redundant network is deployed, failures
can happen. Therefore, it is critical to follow a
proactive, predictive and pre-emptive approach
that identifies incidents before they impact users.
This can be achieved using extensive telemetry
to monitor the different components in the NFV
network. Moreover, this complexity and scale
requires fault conditions to be identified by network
probes with management tools in place that
should be responded to by pre-programmed event
responders (autonomics – big data analytics. This
allows the creation of algorithms to perform tasks
within seconds that may otherwise take operations

Active/Stand-by
Distributed
Pooling engineers hours or days to resolve.
Design
2. Networks should be deployed in a way so that 1. When large scale failure does occur, emergency
Quick Service Recovery
N+M Redundancy failure in a node will not impact service continuity. handling capabilities are needed to recover the
This can be achieved with distributed design across services quickly and minimize impact.
Contingency Plan
multiple sites, deploying additional resources
HA1: VM N+M Redundancy Resource Security 2. Organizations must ensure staff have the right
and proper dimensioning. Also, it is important
Cloud OS

Clustering Hardening
skills as well as the appropriate tools and right
Demarcation capabilities to consider a feature deployment and system
processes in place. They must also ensure
HA2: VM Migration configuration that can protect a network from
network management capabilities meet the
Fault Detection different threats. It is desirable that the network is
same dependable level of carrier-grade reliability
Dimensioning Resilient Design able to identify events or drivers in the network as
expected by customers.
COTS

HA: Multiple NICs, SPs, RAID Automation well as external to the network that could degrade
or otherwise impair the quality or availability of
the supplied services. For example, if a network
Solid plan involving different components and solutions to build a bridge received a significant increase in traffic beyond the
level it was engineered to expect.

NOTE: ETSI NFV ISG (Industry Specification Group)

Carrier Grade has provided some guidelines and best practices on
Enterprise (IT)
Reliability product reliability and resilient network design [27].
Reliability 99.9%
99.999%

Source: Huawei

24 25
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

5.2 Carrier Grade Service – Requirements for 5.3 NFV Reliability Assessment Standard
NFV Reliability In order to provide an ultra-reliable, highly secure Figure 12: Carrier-Grade NFV Reliability Assessment Framework
As illustrated in the diagram below, there is a daunting and fully connected network, an in-depth and
list of requirements for achieving carrier-grade comprehensive NFV network assessment framework
Network Resource Network Network Network Operations &
reliability and deliver the always-on connectivity has been developed. 6 Dimensions
Health Capacity Protection Evolution Handling Maintenance
expected by service providers and their customers.
Such reliability is attained by complying with a very
Hardware Software Overload Inter- Network Staff /
stringent criteria of availability, security, performance, Health Licensing Protection operability Information Organisation
serviceability and management requirements, which (Example: Skills
Design and
fall into six primary dimensions (6D) for a NFV-driven Virtualisation
Physical Autonomous Zero
Topology)
carrier-grade system. Layer Health Response downtime Tools/
Resources
Mechanism Software Platforms
(Example: Upgrade Emergency (Example:
VNF Health Virtual Case Network
360° Link
Machine Processes Telemetry)
Assessment Utilisation Deployment
Elements/ Migration) of new
Figure 11: Requirements for Carrier-Grade NFV Reliability Formulas Connectivity function, Disaster
feature, Recovery Processes
Scaling Security technology Capabilities

Grey Failure
Hardware
Load
Resiliency upgrade and
Balancing
expansion
MANO

Carrier-Grade
Reliability Alarms/
Input Design Statistics Configuration Topology Processes Features
Log-files

Source: Huawei
Availability, Performance,
Security, Serviceability,
Management 1. Network Health
NFV brings a lot of new challenges to operators due
to its architecture. With regards to network health,
operators need to consider several new elements
NFV 6D Assessment Standard when compared to PNF (physical network function)
Network Health, Resource Capacity, Network Protection, that should be in a healthy status, otherwise they
Network Evolution, Emergency Handling, Operations & Maintenance may impact a network’s reliability. Also, if there is
(Measurement, Results, Gap Analysis, Benchmarking, Improvement) deterioration in a basic service KPI then it should be
treated quickly and automatically, to minimize the
downtime or eliminate the impact on users. Some
of the critical elements which should be monitored
Source: Huawei
regularly and proactively include hardware and
software components, the virtualisation layer as
well as connectivity and network performance.
Predictive fault detection becomes more important
in NFV, because applications use infrastructure
delivered by other vendors. For example, operators
should identify grey failures in advance, such as a
memory leak or CPU overutilization (based on a
traffic model) that have not yet affected the service
performance and end users.
2. Resource Capacity
How efficiently and effectively resources are utilised
(e.g. software, hardware, links) is critical for NFV
network reliability. Operators need to monitor and
avoid overloading resources above the network’s
designed thresholds and maintain load-balance
between the resources. NFV introduces scaling
capabilities in order to enhance a network’s agility
and reliability.
3. Network Protection
Networks should be protected from all possible
threats which can impact services. NFV brings a
number of new challenges such as cloud computing,
component scalability and resiliency, information,
cyber security and network access provisioning. A
network should be protected from known threats,
such as signalling storms (overload protection),
denial of service attacks and hardware or software
failures. A network should be designed properly for
its resilience (no single point of failure) and should
be able to recover automatically when possible
(migration, reconstruction etc.). A robust network
shouldn’t be vulnerable to any of these threats.

26 27
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

4. Network Evolution 6. Operations & Maintenance 5.5 Conclusion

A network is always evolving with the introduction
of new features, functions, services, technologies,
software upgrades and hardware equipment etc.
NFV increases the complexity further with a larger
number of components and vendors. A reliable,
high-performance network might be impacted after
a change in one of the layers. For example, when a
vendor’s hardware is incompatible with some of the
components for a server capacity expansion which
creates interoperability issues impacting network
reliability. As a result, openness is critical in driving
innovation and enhancing reliability. Open interfaces
(APIs) play a vital role in building carrier-grade
services. Additionally, it is important to introduce
software upgrade procedures with zero downtime
in order to minimize the impact to end users.
5. Emergency Handling
Failures may happen any time in the network.
Quick recovery and service continuity are very
important, so emergency handling capabilities,
including skills, processes, information (network
topology) and automation, are essential. That is
important for physical networks, but in NFV we
need to consider recovery in this more complex,
multi-layer environment.
In order to maintain high reliability, operational
efficiency is critical. ICT operational transformation is
taking place in operator organizations to adopt NFV
technology. Service providers need to build teams
with skilled staff for new technologies and products,
efficient tools for monitoring (e.g. telemetry),
fault (e.g. root cause analysis) and performance
management, processes for routine maintenance
considering the dynamic environment of NFV
networks. For example, an added complexity in NFV
is driven by the implementation on an as needed
basis. Historically once an application was put into
the network it stayed in the network. Going forward
applications will be applied to the network as needed
then all or a part of them will be removed. In this
situation, the configurations of the network will be
changing on an ongoing basis.
5.4 Best Practices
5.4.1 Use case scenarios
All actors in the industry are invited to use the above
framework to build use cases to get carrier-grade
reliability results, to maintain openness, and to avoid
vendor or technology lock-in. At this juncture, China
Mobile and Huawei have announced partnerships to
carry out the laboratory testing on top of live systems.
Carrier-grade is demanding, but it is an essential
feature of today’s telecom networks to maintain
strict reliability requirements as both the enterprise
customers or consumers have been conditioned to
expect extreme reliability in our networks. However,
with careful planning and the assistance of telecom-
system engineering experts, NFV network and service
designers (new versions of service software will
be necessary to fully utilise the new opportunities
of virtualisation), service providers can build this
capability into their NFV deployments and then, with
carrier-grade systems to confidently commercialize
their NFV services, knowing that they will meet
business and technology objectives while satisfying
their customers.
Service providers know that they need to continue to
meet those expectations as they transition to NFV.
Without this assurance for NFV, they run the risk of
losing their high-value customers and seeing increased
subscriber churn which could offset the many business
benefits provided by NFV. No new technology is
worth that risk, regardless of the potential saving in
CAPEX and OPEX. So, in order to provide an ultra-
reliable, highly secure and full connected network, a
robust, 360 degree, in-depth analytical NFV network
assessment framework is required.

Figure 13: Carrier-Grade NFV Reliability Assessment Methodology

Input 6 Dimensions E2E Assessment Assessment Results

Statistics Network
Health
Resource
Configuration Capacity
Assessment Systems

O&M Resource
Network Capability Capacity
Alarms/Log-files Health Network
Protection
6 Dimensions
Design and their 0

O&M elements 20

Capability 40
Topology Emergency 60
Handling Network 80 Network
Processes Network Evolution Protection
100
Evolution
Emergency
Features
Handling

Source: Huawei

28 29
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

6
Migration

30 31
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
6.1 Introduction
This section describes migrations and software
upgrades from both a deployment and software
upgrade aspect after the l aunch of a virtualised
network launch. Operators should consider
interoperability between entities comprised of PNF
and VNF. Not all network functions are well-suited to
be virtualised and in this section, we examine three
categories from an operator perspective.
6.1.1 Migration from physical network to virtualised
network
This category considers several migration paths from
physical to virtualised, e.g., single functionality level
virtualisation, PNF/VNF-mixed operation in the unique
functionality (e.g., the operation which includes vMME
and existing MME in the same pool), operation without
utilizing NFV orchestrator etc.
6.1.2 Software upgrades
Several aspects should be considered in this category,
for example, VNF application software upgrade, NFVI
(e.g., host OS/hypervisor) upgrade and VIM upgrade
(e.g., OpenStack). There are also some other important
differences in software upgrade procedures between
NFV and traditional networks.
1 – Process
The software architecture is different for each VNF, and
so consequently, the software procedure should be
adjusted accordingly. It is recommended that the VNF
structure and number of VNFCs be considered, before
upgrading the VNF. Another difference between NFV
and a traditional network is that in the NFV it is possible
to access a shared pool of resources (i.e. servers) that
can be used when a process needs them. This provides
flexibility to operators or vendors who could apply a
different strategy based on their needs.
2 – Zero downtime
Each VNF/VNFC may be composed of several
instances and this could provide more flexibility
to operators in order to perform a zero-downtime
software upgrade.
32
3 – Vertical Interoperability Verification
In NFV, there are multiple components provided
usually by different vendors. Each layer’s software has
to be upgraded or updated periodically. Any change to
any layer may impact the interoperability and therefore
the service performance. Before implementing a
sotfware upgrade in the network, it is important to
verify it in an operator’s test bed (or any other external
LAB), mirroring the exact environment with the same
components end-to-end. Once it is verified, the
software release could then be safely rolled-out in the
live environment.
4 – Software upgrade/update frequency
Vertical layout may be composed by different vendors
and each vendor may provide software releases
(patches or versions) in different time periods. Some
vendors may release software packages once a month,
some quarterly or semi-annually etc. Each time, a
multi-vendor verification process may be needed,
which is time consuming and requires additional
resources. Operators should plan accordingly and
consider how to optimize the process per case.
6.1.3 Interoperability
This category considers the interoperability between
the physical entity and the virtualised entity or
network connection using virtualised networking i.e.,
SDN in the NFV configuration.
6.2 Best practices
6.2.1 Migration
Two possible approaches to introduce virtualised
products in an operator network following the launch
of commercial LTE services are discussed below.
1. Introduction of a new system for newly developed
services (e.g., IoT services, facilities for MVNOs and
services for the enterprise customers);
2. Expansion to existing facilities
As the introduction of new systems for newly
developed services is not necessarily a migration, the
rest of this section will focus on area of expansion.
CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
As an example the case of MME deployment is 6.2.2 Software upgrades
discussed here. In this example, the operator who has The types of NFV software to be upgraded are
launched commercial LTE services is going to migrate identified in [11]. These are as follows;
its native MME to a virtualised one. In general, the • VNF domain software
operator has to keep the existing native MMEs operating
• MANO domain software
so that it can continue to serve existing users, therefore
the operator would initially introduce the virtualised • NFVI software
MMEs as additional facilities. In this example, newly From those components, the most critical one is the
added vMMEs are added to an existing MME pool. VNF, since it provides the services. The process should
ensure a zero-downtime software upgrade in order
After a certain period of time, the EOSL (End of
Service Life) of the existing native MMEs is reached, to provide high availability (including planned and
the operator may want to introduce the Next unplanned downtime) and improve a user’s experience.
Generation core network (NGCN) to access one of its The NFVI software upgrade process is not as complex
unique functions e.g. network slicing. In this example, as VNFs, but the process should protect service
the deployment approach is described as a combo- continuity. Finally, the MANO software upgrade process
node deployment of both EPC and NextGen core. can not impact the service, but only operations (e.g.
commands, auto-scaling may not work etc.).
There are several different processes to be followed for
software upgrades based on industry best practice.
Figure 14: MME migration path For each component, the same or different strategy
may be executed and this depends on:
4G 5G • Software architecture
• Availability of Additional resources
Native • Traffic migration capabilities
MME
• Network Design
Addition
to an Virtualised
MME
existing Figure 15: VNF Architecture
pool
Each VNF may be
VNF1 composed of 1 or
EOSL multiple VNFCs
EPC + NextGen Core
VNFC1 VNFC2
VNFCI VNFCI VNFCI VNFCI
Source: KDDI
1 2 N 1
Each VNFC may have 1 or multiple instances (parallable)
Source: Huawei
33
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
1. Using VNF in Pool, so it is easier to migrate traffic
from one VNF to another one. Then upgrade the
“empty” (not loaded) VNF without risk.
2. Rolling SW Upgrade (instance by instance), it is
applicable when there are multiple instances. It is used
for Active-Active or Active Stand-By mode, to isolate an
instance, upgrade it and then put it back to the traffic.
3. Scale-out/Scale-In process, when deployment of
VNFC instances with the new SW (in the same VNF),
migrate traffic to them and kill old instances (scale-in).
Additional resources are needed.
4. New VNF deployment, initially instantiate a new
VNF, migrate traffic to that one and then terminate the
“old” VNF. Traffic migration may require configuration
from external nodes.
5. NFVI SW Upgrade (i.e. host OS, KVM, Open V
switch or DPDK) uses live VM migration to minimize
the impact. VMs are migrated from NFVI (old SW) to
the upgraded NFVI area and continue upgrading all
the resources.
6. MANO SW Upgrade process is not as critical as
other components, since there is usually no service
impact. Any method highlighted above could be used
based on the SW architecture.
34
CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
6.3 Requirements
7
6.3.1 Migration
The following aspects should be considered for the
migration from physical to virtualised:
• It should be possible to share PNF and VNF
resources for the same network function
NOTE: The “resources” are not the resources which
are provided by the NFVI, such as compute, storage
or network resources
• In a mixed environment consisting of both physical
Performance benchmark
and virtualised network functions, it should be
possible to maintain the same interfaces between
network functions
for NFV infrastructure
6.3.2 Software upgrades
The following elements should be considered for
software upgrades;
• The traffic should be switched seamlessly for all the
methods of SW upgrade described above
• Migration of active VNFs (live migration) from the
compute node in operation to the maintenance zone
should be executed without interruption when VIM
software or NFVI software will be upgraded. If the
interruption occurs, it should be minimized
• Required hardware acceleration functionality should
be available for the destination of the migration
• Due to complexity, automated software upgrade /
update process is preferred
• Quick roll-back process in case of failure
• Ensure zero downtime for the services during
VNF upgrade
35
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

7.1 Performance benchmark scope
When NFV was brought to the industry, it was claimed
to help operators shorten their service time to market
and offer agility in system expansion, while simplifying
network operation through the use of common
servers. However, since NFV is built on common x86
servers, service level performance is often questioned.
Given the circumstances, vendors and operators
should team up to provide some performance metrics
and carry out tests.
Each single portion of the NFV system can be
benchmarked, for example, VNF performance, NFVI
performance, storage performance, virtual switch
performance etc. Each level of these performance
benchmarks will help operators to decide which
product will fit into their NFV systems. From the end-
to-end service perspective, it is possible to benchmark
service deployment. For example, how quickly a NFVI
could be set up, how long it would take to upload a
VNF package and how long to instantiate a VNF or
a Network Service etc. Based on these benchmarks,
operators can tell if it is feasible to deliver a low cost,
high performance solution based on NFV.
In addition to the vendors and operators’ own
tests, many standard organizations and open
source communities are very supportive of NFV
performance benchmarking experiments. Usually
this work will involve test framework and traffic
generators, as benchmarking could not be performed
in live networks. Other common characteristics of
performance benchmarking work are as follows:
• Include testing instructions that are sufficiently
specified (prerequisites, procedures, output)
• Results are repeatable
• Test cases can be performed on different vendor’s
device/system
• The produced results are useful and meaningful for
the users
In this chapter, state of the art NFV performance
benchmarking is outlined. Performance metrics, test
methodologies and use cases that have been defined
will be helpful for vendors and operators who want to
carry out the tests in their own labs.
7.2 State of the art in NFV performance benchmarking In the OPNFV Colorado release, test cases covering
In OPNFV, there are several test projects which have Performance/Speed and Capacity/Scale are available,
been initialized to benchmark the performance of Reliability/Availability part will be finished in the
a NFV system and its components. Yardstick and coming releases.
Bottlenecks are the names given to projects used
The scope of Yardstick is the development of a test
to benchmark system level performance, while
framework as well as test cases and test stimuli to
StorPerf, VSPerf and CPerf define methodology
enable NFVI verification. This methodology is aligned
in benchmarking storage, virtual switch and SDN
with ETSI TST 001 [18].
controller performance respectively.
The Bottlenecks [13] project aims to find system
Yardstick [12] is a project that aims to verify the
bottlenecks by testing and verifying the OPNFV
infrastructure compliance from the perspective of a
infrastructure in a staging environment before
VNF. Based on the uses cases that have been defined
committing it to a production environment. It defines
in ETSI GS NFV 001 [17], each use case implies specific
an automatic method for executing benchmarks to
requirements and complex configuration on the
validate the deployment during the staging phase.
underlying infrastructure and test tools. In order to find
The framework has four components: Workload
a system level benchmark, in Yardstick, every single
generator and VNFs (WV), Monitor and Analysis (MA),
VNF work-load performance metric is broken down into a
Deployment and Configuration (DC), Automated
number of characteristics/performance vectors and each
Staging (AS). The architecture is shown as follows:
performance vector is then represented by a test case.

Figure 17: YardStick performance metrics and test suite

Figure 16: NFV performance benchmark scope Performance/Speed Capacity/Scale Reliability/Availability
Compute • Latency for random memory access • Number of cores and threads • Processor availability (error free
• Latency for cache read/write • Available memory size processing time)
operations • Cache size • Memory availability (error free
Projects Working Groups Test Framework SUT/DUT/FUT for NFV • Processing speed (instructions per • Processor utilisation (max, average, memory time)
second) standard deviation) • Processor mean-time-to-failure
• Throughput for random memory • Memory utilisation (max, average, • Memory mean-time-to-failure
access (bytes per second) standard deviation) • Number of processing faults per
ETSI Sufficiently Defined VSwitch
• Cache utilisation (max, average, second
• TST-001 Instructions
standard deviation)
• TST-002 Controllers
Test Framework Network • Throughput per NFVI mode (frames/ • Number of connections • NIC availability (error free
• PER-001
+ byte per second) • Number of frames sent/received connection time)
• IFA-003 Repeatable Results Storage
Traffic Generator • Throughput provided to a VM • Maximum throughput between VMs • Link availability (error free
(frames/byte per second) (frames/byte per second) transmission time)
OPNFV
NFV1 • Latency per traffic flow • Maximum throughput between NFVI • NIC mean-time-to-failure
• Yardstick Applicable to Different
Vendors • Latency between VMs nodes (frames/byte per second) • Network timeout duration due to link
• Bottleneck
Management • Latency between NFVI nodes • Network utilisation (max, average, failure
• StorPerf
• Packet delay variation (jitter) standard deviation) • Frame loss rate
• VPerf
Benchmarking Useful and Meaningful Reliability between VMs • Number of traffic flows
• CPerf
Results for Users • Packet delay variation (jitter)
Servers between NFVI nodes
IETF (BMWG)
Test Traffic Storage • Sequential read/write IOPS • Storage / Disk Size • Disk availability (error free disk
• Virtual-Net-04
Framework Generator Unambiguous Output • Random read/write IOPS • Capacity allocation (block-based, access time)
• HA-NFVI-01 VNFs
• Latency for storage read/write object-based) • Disk mean-time-to-failure
operations • Block size • Number of failed storage read/write
• Throughput for storage read/write • Maximum sequential read/write IOPS operations per second
Source: Huawei
operations • Maximum random read/write IOPS
• Disk utilisation (max, average,
standard deviation)

36 37
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
Figure 18: Bottlenecks test framework
Workload
generator
VNF
Hypervisor ODL DPACC
Infrastructure
Source: OPNFV Bottlenecks project
The workload generator generates workloads which
go through VNFs, and then analysis units will monitor
the infrastructure and VNF status. Test results will
then be analysed to find out the system bottleneck.
Therefore, a wide range of different hardware
resources and software configurations will be used to
locate a system’s bottleneck.
The Storage Performance Benchmarking for NFVI
(StorPerf) [14] is a project that aims to provide a tool
to measure block and object storage performance in
an NFVI. The project will define a test suite, including
test cases, metrics and test process to find the
benchmarks, which can provide a good preview of
expected storage performance behaviour for any type
of VNF workload.
The vSwitch Performance (VSPerf) [15] project will
develop a generic and architecture agnostic vSwitch
testing framework and associated tests, which will
serve as the basis for validating the suitability of
different vSwitch implementations in a Telco NFV
deployment environment.
The Controller Performance Testing (CPerf)
[16] project will serve as a performance testing
environment for the SDN controller portion of the
large, realistic, automated deployments required by
OPNFV. The initial focus of CPerf is OpenDaylight, but
later-on the project tends to leave the controller part
of the test matrix open and will welcome collaboration
with other controller communities.
38
Automated
Staging (AS)
VNF Monitor and
Analysis (MA)
WV
OpenStack
Deployment and
Configuration
(DC)
The Network Service Benchmarking (NSB) is an open
source framework that allows benchmarking and
characterization of VNFs by testing with real-life traffic
scenarios on Bare Metal, Standalone Virtualised, and
Managed Virtualised environments. The NSB test
harness is inherited from the OPNFV Yardstick open
source solution (also upstream the patches back to
Yardstick community developed in test harness),
and incorporates a benchmarking methodology that
facilitates deterministic and repeatable benchmarking
on Industry Standard High Volume (SHV) servers. It
provides several VNFs: vCG-NAPT, vACL, vFW, vPE,
vEPC as the samples for demonstration, and it also can
integrate 3rd-party VNF as the benchmarking tool for
operators, which can help in characterizing VNFs by
measuring Network, VNF and NFVI KPIs:
• Network KPIs: e.g. Traffic Generator KPIs like
packets in, packets out, throughput, latency as per
RFC 2544 etc.
• VNF KPIs: e.g. packets in, packets out, packets
dropped, etc.
• NFVI KPIs: e.g. CPU utilisation, IPC, L1/L2/LLC
cache hit/misses, iTLB/dTLB hit/misses, OVS stats,
memory bandwidth & latency, etc.
In ETSI, ETSI GS NFV-TST 001 [18], TST 002 [19], PER
001 [20] and IFA 003 [21] have been produced by ETSI
Industry Specification Group (ISG) to benchmark the
NFV-related performance and test methods.
CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
TST 001 provides an informative report on methods for
pre-deployment testing of the functional components
of an NFV environment, including VNFs, NFVI and
MANO. The recommendations focus on lab testing and
the following aspects of pre-deployment testing:
• Assessing the performance of the NFVI and its
ability to fulfil the performance and reliability
requirements of the VNFs executing on the NFVI
• Data and control plane testing of VNFs and their
interactions with the NFV Infrastructure and the
NFV MANO
• Validating the performance, reliability and scaling
capabilities of Network Services
TST 002 provides interoperability test methodology
that is applied to NFV by analysing some of the core
NFV capabilities and the interactions between the
functional blocks defined within the NFV architectural
framework required to enable them. It describes
two types of testing: Conformance Testing and
Interoperability Testing.
PER 001 provides a list of minimal features which
the VM Descriptor and Compute Host Descriptor
should contain for the appropriate deployment of
VM Images over an NFVI, in order to guarantee
high and predictable performance of data plane
workloads while assuring their portability. In addition,
the document provides a set of recommendations
on the minimum requirements which hardware and
hypervisor should have for a NFVI suitable for different
workloads (data-plane, control-plane, etc.) present in
VNFs. According to the workload analysis in clause
6, clause 7, it provides a recommendation on the
minimum requirements that a Compute Host should
have for a NFVI suitable for data-plane workloads, and,
clause 8 gathers the list of features which the Compute
Node Descriptor and the VM Descriptor templates
should contain for the appropriate deployment of VM
Images over an NFVI, in order to guarantee high and
predictable performance while preserving portability
across different servers.
IFA 003 specifies performance benchmarking metrics
for virtual switching, with the goal that the metrics
will adequately quantify performance gains achieved
through virtual switch acceleration conforming to the
associated requirements specified herein. It defines the
critical aspects of vSwitch performance by treating the
vSwitch as a Device Under Test (DUT), with specific
configurations that are consistent across instantiations
of a vSwitch on a computing platform. It also uses
the existing testing and benchmarks specifications
(see [22], [23], [24] and [26]) to measure the
performance of the DUT under specific configurations
and conditions, such as vSwitch physical to physical,
vSwitch virtual to virtual, etc.
In IETF, the draft-ietf-bmwg-virtual-net-04
[22] document researched by Benchmarking
Methodology Working Group (BMWG) has defined
the considerations for benchmarking virtual network
functions and their infrastructure. This document
investigates additional methodological considerations
necessary when benchmarking VNFs instantiated and
hosted in general-purpose hardware, using bare-metal
hypervisors or other isolation environments such as
Linux containers. It lists several new benchmarks and
related metrics, such as time to deploy VNFs, time to
migrate VNFs, time to create a virtual network in the
general-purpose infrastructure, etc. it also defines
several new considerations which must be addressed
to benchmark VNF and their supporting infrastructure,
like hardware configuration parameters (shelf
occupation, CPUs, caches, memory), etc.
Another document, “Considerations for Benchmarking
High Availability of NFV Infrastructure” [27]
lists additional considerations and strategies for
benchmarking high availability of NFV infrastructure
when network functions are virtualised and performed
in NFV infrastructure. With VNFs and NFVI replacing
the legacy network devices, operators have several
issues to cope with such as availability, resiliency and
non-measurable failures. Above all, they want to
ensure the availability of the VNF products and their
infrastructures. From the operator point of view, the
availability is the most important feature and the
benchmarking tests for the high availability of NFV
infrastructure are also important. This document
investigates considerations for high availability of NFV
Infrastructure benchmarking tests.
39
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

8
Vertical
interoperability

40 41
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

8.1 Introduction
Physical mobile networks are made up by equipment
provided by multiple vendors in general. This is made
possible by the clear definition of open interfaces
between the network nodes specified by 3GPP.
Such a multi-vendor environment fosters innovation
and competition, resulting in advantages both for
operators and subscribers.
It will be beneficial for virtual networks to be made by
equipment provided by multiple sources.
8.2 Requirements for vertical interoperability
Figure 18 highlights an example of a virtual network
where individual components are procured from a
number of different vendors.
For networks to operate correctly it is important that
the interfaces between the individual blocks of the
architecture are tightly specified and implemented.
Failing to define interoperable interfaces is expected
to result in a higher total cost of ownership as well as a
slower time to market. Operators will need to ensure that
the open source community developing the architecture
understand the importance of vertical interoperability.
8.3 Challenges and potential risks
When vendors use non-standardised non-
standardised interfaces there could be potential risks
in the network. Some of the potential impacted areas
are described below:
1. Vertical Integration
With characteristics such as hardware-software
decoupling and hardware generalization, NFV further
opens up carriers’ network architecture. NFV is one
huge complex ICT systems integration project, involving
multiple technologies, interfaces and multiple vendors.
It is important that vertical cloud platform integration
is performed smoothly and quickly. Interoperability
plays a key role in vertical integration, especially in a
multi-vendor ecosystem.
2. Network Service Deployment
Interoperability can cause problems in Network Service
deployment, since multiple components provided by
different vendors are involved during this process. This
includes from OSS/BSS and Portals up to NFVO, VNFM,
VIM, Cloud OS and COTS. Descriptors, such as NSD,
VNFD should also follow the standardised format for a
successful service instantiation.
3. Service Assurance
Service performance and high availability are also
dependent on interoperability. When there is a failure
in the network, the information should be propagated
quickly to the upper layers, based on the standard
interfaces so that each component can use this
information effectively. For example, a hardware
failure should trigger an alarm to users allowing them
to act immediately.
Automation also plays an important role in NFV
lifecycle management and different components have
in-built self-healing mechanisms (i.e. VM migration in
case of host failure). These mechanisms are triggered
under specific conditions, such as notifications that
have been received from the network. If this kind of
information is not received then the system will not
recover automatically, and cause network outage.
4. Software Upgrade
Any change in the software of a component, in any
layer, could cause problems to the other layers.
Consequently, this could impact existing services and
their users. Vertical verification is recommended in
order to test interoperability. Horizontal verification
challenges remain the same as in traditional network,
so there is no real difference.
8.4 Best Practices
Incompatible systems elements or software versions
can cause further network faults, and require
compatibility management. The key elements are to
use standardised interfaces and proper compatibility
validation processes.
1. Standardised Interfaces
NFV solutions should be based on interoperable multi-
vendor ecosystems with open source technologies
or standardised protocols and interfaces. ETSI’s IFA
Workgroup focus on interface standardisation and
several open source projects have developed platforms
based on this, such as OPNFV, OPEN-O, OSM etc. There
have also been several documents released by ETSI.
Figure 20 shows some of the important work done by
the ETSI IFA and SOL Working Groups.
2. ETSI Plugtests
ETSI identified the need for thorough industry
interoperability testing and took on the initiative to
start organizing NFV Plugtests. They will offer test
sessions where vendors and Open Source projects will
be able to assess the level of interoperability of their
implementations and verify the correct interpretation
of the ETSI NFV specifications.

Figure 20: MANO to IFA Mapping – reference points

Figure 19: example of multivendor virtual network
Reference of the functional specification
IFAxxx
IFA013 (stage 2) ETSI GS NFV IFA XXX
Orchestrator SOL005
OSS/BSS NFV Orchestrator vendor Reference of the solutions specification
OSS NFVO SOLxxx
(stage 3) ETSI GS NFV SOL XXX
Os-Ma
EM EM EM EM
REST APIs IFA014
Management & Orchestration

NFV IFA007
SOL003 SOL001
Manager(s) IFA011
VNF VNF VNF VNF VNF Or-Vnfm SOL001 NSD
IFA008
Service, SOL002
VNF and EM & NFV Descriptors
Infrastructure VNFM Or-Vi (templates)
Virtual resources Description
VNF
Ve-Vnfm
Compute Storage Network
VNF IFA011
Execution
Vi-Vnfm Package SOL001
IFA002 VNFD
Virtualisation layer IFA018
IFA006
Hardware Virtualised
Infrastructure IFA004 IFA005
IFA019
Compute Storage Network Manager(s)
NFVI VIM
Open Stack APIs TOSCA/YAML Profile
Nf-Vi (extensions required) +CSAR archive format
Infrastructure vendors
Source: ETSI NFV

42 43
 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK

The component for the test sessions are Virtual Network

Functions (VNFs), Management and Orchestration
(MANO) solutions and NFV Infrastructure (NFVI) with
pre-integrated Virtual Infrastructure Manager (VIM).

3. Integration and Verification

Most of the interoperability issues appear during
the integration phase. Multiple vendors are involved
in this phase, so cooperation and coordination are
important. Either operators or one of the assigned
services suppliers should lead this complex project
and act as a prime system integrator, having
extensive IT and CT experience, in order to deliver
an integrated E2E solution.

A proper verification plan is required for interface

compatibility and service validation. For this purpose, an
operator’s test bed could be used, using the same multi-
vendor ecosystem. Alternatively, a vendor’s NFV Lab
could be used for solution validation or pre-packaged,
pre-integrated and pre-tested solutions. The same
process applies for a software upgrade process.

44
 Find out more at
www.gsma.com/network2020

GSMA HEAD OFFICE

Floor 2
The Walbrook Building
25 Walbrook
London EC4N 8AF
United Kingdom
Tel: +44 (0)20 7356 0600
Fax: +44 (0)20 7356 0601