LightReading

www.lightleading.com

REPORT
MARCH 2015

TESTING
PARTNER:

PARTICIPATING
COMPANY:

Validating Ciscos Service Provider

Virtualization & Cloud Portfolio

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading

REPORT

TESTING
PARTNER:

MARCH 2015

PARTICIPATING
COMPANY:

contents

www.lightleading.com

Introduction
EANTC Heads to San Jose
Step 1: The CSP Evaluation and Proof of Concept Stage
Step 2: The Deployment Stage Cisco Service Provider SDN Deployment Tools
WAN Automation Engine: Maintenance Planning
Cisco WAN Automation Engine Disjoint Tunnel Creation
Cisco WAN Automation Engine Bandwidth On Demand
Cisco WAN Automation Engine Bandwidth Calendaring
Cisco WAN Automation Engine Tunnel Split/Merge Manager
WAN Automation Engine Summary
Cisco Network Service Orchestrator enabled by Tail-f
Cisco Network Service Orchestrator Service Provisioning
Cisco Network Service Orchestrator Service Modification
Cisco Network Service Orchestrator Service Restoration
Cisco Network Service Orchestrator Service Restoration Due to Node Failure
Cisco Network Service Orchestrator Device Management
Cisco Network Service Orchestrator Service Model Modification
Cisco Network Service Orchestrator (NSO) Tests Summary
Step 3: The Cloud Services Creation and Delivery Stage, Cisco Cloud Managed VPN Solution for Virtual Managed Services
Cisco Cloud VPN: Behind the scenes
Cisco Cloud VPN Solution Life Cycle Conclusion
Production Environment Application Demonstrations Project Squared and Mobility IQ
Conclusion: Meaningful Developments

02

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

Introduction

many of the same techniques were used in this engagement with Cisco,

The big question facing every communications service provider (CSP)

be evaluated for performance but instead with a more holistic service-

right now is: How can I build and run a programmable, intelligent,
responsive, efficient, flexible, secure yet open network that has a
high degree of automation, enables me to configure and activate new
services quickly, makes best use of emerging cloud capabilities, meets
customers needs and helps me make money?

EANTC found itself faced not so much with a set of boxes that needed to
enabling architecture and environment that could be functionally
explored and put through its paces.
What they found is that a number of complex provisioning and fault
management aspects of service provider networks can be automated
and programmed on demand, and that services can be rolled out

In British soccer circles, thats called a big ask.

much more quickly and easily than with traditional configuration and
activation tools.

Yes, its a major challenge but its not insurmountable. There are
many smart innovators in the global communications and networking

The following pages, then, deliver a detailed account of the EANTC

technology sector working on the answers and coming up with

teams processes and findings. We hope you find the report

solutions every day. What CSPs need to know is whether these new,

informative and useful.

next-generation, New IP technologies can meet their needs right now

and in the future. As an independent and trusted media organization at

T
 he Light Reading team and Jambi I. Ganbar, Senior Technical

PARTICIPATING
COMPANY:

virtualization (NFV), software-defined networking (SDN) and cloudbased applications offerings.

For this reason, Cisco provided EANTC with access to various tools, labs
resources and other capabilities to, in essence, play the role of a CSP that
is offering business VPN services to its customers.
Ciscos goal was to take us on a three-stage journey very similar to that
being taken by many CSP customers that are creating and delivering
business VPN services.
Step 1:
The CSP Evaluation and Proof of Concept Stage
CSPs can begin their journey by experimenting with web portals and
using a cloud in a box tool that helps them get to grips with the
possibilities of programming a virtual, elastic environment.
Step 2:

the heart of the global communications technology community, Light

Marketing Manager and Carsten Rossenhvel, Managing Director,

Reading is best placed to provide some critical answers to these all-

European Advanced Networking Test Center AG (EANTC) (http://

important questions.

www.eantc.de/), an independent test lab in Berlin. EANTC offers

Once such experiments are over and CSPs understand the possibilities,

vendor-neutral network test facilities for manufacturers, service

Cisco can then support their customers with development tools that

providers, and enterprises.

will help them create the cloud solution that best suits them, as well as

As part of that role, Light Reading asked another independent and

products that enable CSPs to deploy network capacity and services.

trusted organization, the European Advanced Networking Test Center

(EANTC), to visit Cisco Systems in San Jose, Calif., put itself in the
shoes of a CSP and conduct a series of validation and verification
exercises on a number of Cisco cloud, software-defined networking
(SDN) and virtualization platforms, which Cisco has been developing
during the past few years.
The EANTC team spent two weeks at Ciscos premises, conducting
multiple tests, asking lots of important questions and keeping track of
what they did, why they did it and what the outcomes were.
This report is the result of those two weeks. On the following pages
you will find the EANTC teams blow-by-blow account of how they
analyzed, verified and used Ciscos technology in the same way as a
network operator team, and the results of their findings.
Whats important to note is that the EANTC team adapted its modus
operandi in a way that was relevant to the cloud environment it
encountered. EANTC is known for testing the attributes, limits and
multivendor deployment suitability of networking technology. While

03

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

The Deployment Stage: Cisco Service Provider SDN Deployment Tools

EANTC Heads to San Jose

When Light Reading called, EANTC jumped at the chance to visit Cisco
and investigate its cloud and virtualization service provider solution.
Cisco prepared a comprehensive story for our team to investigate in its
San Jose facilities, with the premise based around two core questions:

Step 3:
The Cloud Services Creation and Delivery Stage
Cisco offers the capabilities to manage and run network services and
applications using cloud-enabled data centers. This is a long-term play
-- a cloud service based on Ciscos expertise in networking and the

H
 ow can communications service providers (CSPs) comprehend the
myriad possibilities the new world of virtualization and softwaredefined networks brings?
How can CSPs develop and offer new services with the same speed
and flexibility as web services/OTT players such as Google, Amazon
and Netflix?

data center. To illuminate the possibilities offered by this service, Cisco

During the course of the verification process we heard Cisco refer to

urgency, agility and time-to-deployment like we never have before: That
sort of talk is normally the domain of software development teams.

Step 1: The CSP Evaluation and Proof of

Concept Stage

Cisco explained that its service provider customers are at different

stages of evaluating, deploying and leveraging their network functions

Specifications Group, which is run by a core group of major telecom

prepared some applications that demonstrated how CSPs can generate

revenues from their cloud platform investments.
First up Step 1.

As a result of EANTCs engagement with ETSIs NFV Industry

operators, we know that the largest CSPs in the world are actively

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT
MARCH 2015

working on accelerating the adoption of virtual network functions,

developing cloud services and adopting web-scale operations.
There are hundreds of other network operators too, of course, and
many (if not all) will be seeking help on how to make their networks and
operations more efficient and flexible. So how can CSPs of all types, no
matter what their resources, engage with Cisco and figure out what the
vendor has to offer in terms of cloud enablement and virtualization?
One entry-level resource is a web-based platform called dCloud
(http://dcloud.cisco.com/), an online resource that is freely available
for anyone to access and use and which, in essence, illustrates what is
possible in the New IP world. To validate this tool, an EANTC engineer
created an account and ran the scenarios himself. Once an account is
created, the user is given a set of pre-defined setups that demonstrate
some of Ciscos newer offerings and collaborations, the latter of which
includes an SDN controller based on OpenDayLight specifications.
In total, we encountered 40 different scenarios. We ran through three
different dCloud scenarios, each of which was a preview of a real
application that we tested later in the two-week process.
The scenarios were:
Cisco WAN Automation Engine 6.0 with 8-Nodes v1
Cisco CloudVPN Technology Preview v2
Cisco Network Control Systems 3.3 MPLS VPN v1
In this phase of the verification, dCloud was used to provide us with
a preview of what we would encounter later. As part of our processes,
we requested that one of the dCloud scenarios (called Sandbox) be
connected to our own laptops for independent verification. This was
easy to enable and we were able to run the scenario via an external
connection from the Cisco Sandbox to our laptop device.
Ciscos dCloud is an enabler designed for two types of user: Cisco
sales engineers and account teams; and customers. It can connect
to public clouds, such as Amazon Web Services (AWS), as well as to a
CSPs own network infrastructure.
If a CSP prefers not to use a public cloud for the evaluation process,
maybe because of concerns related to the security of its systems
and data, Cisco has a solution -- an innovation and development pod
called /dev/innovate.
This innovation pod is a self-contained data center rack that Cisco
leases to CSPs on an annual basis. The pod includes a range of Cisco
products that a CSP can use to build cloud services:

04

TESTING
PARTNER:

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

Cisco UCS 5108 Blade Server Chassis

Two UCS 2208XP blade chassis extenders
Six UCS B200 M3 compute nodes
Two UCS 6248 fabric interconnects
A Cisco ASR9001 router
One Nexus 5672 Data Center switch

The pod also has enough rack space to host any additional
components, enabling the CSP to incorporate additional systems from
Cisco or even from other vendors.
The aim of the pod is to enable a CSPs R&D team to directly interact
with that CSPs operations group in order to address questions around
scalability, security, high availability, and so on. It has other benefits
too, as it could shorten the time needed to get from concept to service
development and creation and bring CSPs closer to a web speed
development timeframe.
All of this is part of Ciscos efforts to help CSPs speed up the
decision-making and development processes as they consider their
virtualization strategies.
The next logical step, following the evaluation stage, is for a CSP to
move to the deployment process.

Step 2:
The Deployment Stage Cisco Service
Provider SDN Deployment Tools

PARTICIPATING
COMPANY:

Ciscos WAN Automation Engine provides a lot of very useful

functionality for a CSP. The tool collects historical network utilization
data and is able to predict future behavior based on past and present
heuristics. In essence, the software is used to manage traffic in CSP
backbone networks. Managing backbone traffic is typically a question
of understanding congestion and congestion-related events, a
powerful concept that ties very well to the fundamentals of SDN.
WAE uses Path Computation Element Protocol (PCEP), as well as
BGP-LS, to learn the network topology and install state in network
elements. To show how it works, Cisco set up a large demonstration
network comprising 35 routers, including the ASR9006, CRS-4/S and
several models of the Cisco 12000 product family. As it is rare that any
network operator would deploy infrastructure from a single supplier,
Cisco also incorporated IP infrastructure from a third-party vendor.
Judging from EANTCs 12-year history of annual multi-vendor
interoperability testing at the MPLS World Congress in Paris, this was
a very positive and meaningful decision.
Armed with the details of what was to be used in the verification
process, we headed to the lab.
Our first task was to disconnect a link between two of the devices
and examine the outcome so that we could be sure that what we were
seeing was a real network setup.

For the deployment of an SDN-enabled service provider cloud, Cisco

has developed its Evolved Services Platform (ESP). This is designed to
provide an agile, automated and reliable operation environment that,
Cisco states, includes predictable design, capacity planning and multivendor provisioning.
The two key elements of ESP that Cisco provided for the test are:
the Cisco WAN Automation Engine (WAE), developed following the
acquisition of Cariden in late 2012;
the Cisco Network Service Orchestrator enabled by Tail-f, which, as
the name suggests, has been developed from technology acquired as
part of the purchase of Tail-f in 2014.

Figure 1: Cisco MATE Live Network Overview (part of Cisco WAE)

The tests we executed with WAE were designed to verify its
functionality as well as demonstrate that web-based applications
could be installed on top of WAE. To achieve, the verification process
was split into two sections: A CSPs offline activities; and live network
operations activities.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

WAN Automation Engine: Maintenance

Planning

PARTICIPATING
COMPANY:

extra protection against packet loss. As the Cisco team noted, this
is particularly important when the content is video and the content
provider does not want to lose any packets in case failure occurs.

Once the tunnel disjoint command ran, we logged back into the routers
to find a very different configuration than the simple one line that was
used to create the LSPs previously.

So, how can a CSP guarantee that two transport paths are diverse
enough so that the content provider is not risking the loss of any
packets in the case of a single link or node failure?

We then waited five minutes and verified that the links were being
used based on the Ixia N2X tester traffic streams we configured in
the network. We checked to see that the configuration was installed
on both Cisco and the third party router. At that point we increased
the rate of the test traffic and monitored that the colors of the lines
changed based on utilization (the tool uses colors to symbolize link
utilization that made for a nice visual effect over time).

In our view, having a system that informs a network operator about

the effect that device maintenance will have on availability and
performance is beneficial. In this functional test, we prepared for a
maintenance activity on the interfaces of one of the routers.
We used MATE Live, which is part of the WAE tool suite, to indicate
that we intend to take an interface (in this case, on a node called RP2)
offline at a certain time. As part of that process, we also factored
in the possibility that a network failure might occur while the RP2
maintenance was underway and requested the application to include
that in its report.
With those requests submitted, the system reported back with a
long list of consequences resulting from the combination of the

In essence the WAE software function was clear: Verify that two MPLS
transport tunnels, entering the network at different locations but
exiting the network at the same location, do not share any link or node
in the path.
To verify that this function works, we used the MATE tool to build a
pair of LSP (label-switched path) tunnels between the R30 and R35
nodes (the latter node being a third-party router, not a Cisco device).
We generated test traffic and verified via device CLI (command line
interface) that the traffic was passing over the two tunnels.

maintenance and additional node failure. In bold green letters the

MATE Live reported PASS no congestion requirements violated.
We received a detailed list of the resulting connections to alternative

Figure 4: The resulting disjointed LSPs displayed by Cisco MATE Live

devices, with the software predicting that the link utilization on the
following routes would be up to 98.7% but would not exceed 100% in
any instance.
With those details as our guide, we decided to put the network, and the
MATE Live softwares predictions, to the test. The Cisco team informed
us that initiating the procedure would generate a paper trail that
would show whether MATE Live provided an accurate prediction.

Figure 2: Dual Tunnel Setup using Cisco MATE Live

Next we used the tunnel disjoint operation to split the Label Switched
Paths (LSPs) we just built.

So while test traffic was running in the network, we entered the lab
and took down the identified RP2 interface to disturb the networks
topology. We then compared the bandwidth predicted by MATE Live
with the actual bandwidth and found them to be almost identical
EANTC -- the application was only 1.04 Mbit/s out).

Cisco WAN Automation Engine Disjoint

Tunnel Creation

05

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

The function that MATE Live provides is the equivalent of a what if

function. Once the CSP is confident that a change should take place
and confirms using the MATE Live GUI, NSO kicks in and sends the
configuration changes to the routers.
We noted, by comparing between pre- and post-configuration
data that the post-tunnel disjoint operation was significantly more
complicated -- in terms of configuration its easier for the system
to set up a path than to tear it down. But the CSP -- in this instance,
our EANTC test engineer -- did not actually have to spend any time
creating the device configuration.

(591.06 Mbit/s predicted compared with 590.02 Mbit/s recorded by

For a few mission-critical applications, CSP networks are sometimes

configured to transmit two copies of the same traffic streams for

We were interested in the mechanics and asked how MATE Live

actually created a configuration on the routers. Cisco explained that
the system interacts with the Cisco Network Service Orchestrator
enabled by Tail-f using a standards-based API. MATE Live uses a
model-processing tool that sends instructions to the Network Service
Orchestrator (NSO), which in turn connects to the devices.

Figure 3: Disjoint LSP operation using Cisco MATE Live

But before we dive deeper into NSO, we still had other WAE functions
we needed to verify.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

PARTICIPATING
COMPANY:

Cisco WAN Automation Engine

Bandwidth o n Demand
The next verification process involved a set of web-based applications
that run on WAE.
What is the relationship between a web browser application and the
WAE software? As explained on Ciscos DevNet, WAE exposes REST
(Representational State Transfer) application programming interfaces
(APIs) that enable developers (in this instance from Cisco) to develop
applications that can run in a web browser.

Figure 5: Service Path Utilization in the Network

Figure 7: Bandwidth on Demand Recommended Path

We used the web browsers developer view to capture the interaction

We then used the bandwidth on demand tool to request an additional

400 Mbit/s for our service. The application provided the report as
shown in the screenshot below. As you can see, the system reported
congestion on the first hop in our path (other traffic was running in the
network as one would expect in a CSP network).

As Figure 7 shows, the system created a complete new path that

actually took a longer route from A to Z and arrived at the end point
(marked by Z on the topology) from an unpredicted direction (to our
eyes, it arrived from the rear).

between the browser and the application and confirmed that REST
communications with WAE were being made.
The first application we tested was Bandwidth on Demand. This tool
allows the CSP to dynamically add service capacity between network

The number of nodes in our path changed from 5 to 7, but by using

this path, both LSPs, as the colors indicate, predicted less than 50%
utilization. Once we accepted the configuration, we generated traffic
again, this time at the cumulative rate (762 Mbit/s plus 400 Mbit/s)
and saw the links on the topology change their colors to light green,
an indication that no link was congested. We also recorded zero traffic
loss, which was what we expected.

nodes, even, as Cisco explained, for just a short period of time.

We immediately identified the benefit to end users of this capability
since we occasionally host large groups of vendor engineers during
EANTCs public interoperability events, which last for a few weeks.
During this time, our network connection becomes congested and, as
a consequence, it slows down considerably. If we could use a web tool

As one of the Cisco managers who attended the process noted, this
is a little bit like an autopilot on a plane -- you still need a pilot that
really knows what he is doing on the plane to operate it.

to add bandwidth for those weeks -- and only for those weeks -- all
these engineers we are hosting will enjoy faster download times and
will have an easier time collaborating with their developer colleagues
who are often located in faraway locations.

Figure 6: Bandwidth on Demand Utilization Report

To test this functionality, we created a service (in this case, an MPLS

network operator is merely supervising the system.

Label Switch Path), between two nodes (R35 and R30) and sent traffic

We also received an option that prompted us to optimize. Once

at 762 Mbit/s. Since the link capacity in the lab was 1 Gbit/s for the

we selected this option, the system recommended a new path (as

majority (but not all) of the path, the GUI displayed the link utilization

displayed below in Figure 7).

in yellow for the GigE links and green for the 10GigE link.
Since we did not specify any hop limit or specific latency, we were
The colors signify certain levels of network utilization and are used

curious to see how the application intended to provision additional

to warn CSPs of the likelihood of congestion: The CSP can set the

capacity for our service. The solution was a network route that most

thresholds, with green representing utilization below a certain level

likely would not have been selected by a human engineer, with the

and yellow above a certain level. The system can also display links in

brown arrows signifying the what if routes suggested by the tool:

red, which signifies that an even higher threshold has been breached.

06

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

This indeed is a novel approach to network engineering, where the

system is responsible for developing solutions to problems and the

Cisco WAN Automation Engine

Bandwidth Calendaring
If an application has an extensive database of past and present
network states -- one that includes link utilization, latency, and device
layout -- the application could attempt to calculate (or predict)
future utilization of the network.
The next application Cisco demonstrated for this purpose is called
Bandwidth Calendaring. This application enables a service providers
customer to reserve bandwidth for a limited period of time in the future.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

EANTC played the role of the enterprise in this process by using a web
interface to order more bandwidth. Such a capability is a great way
to facilitate speedy off-site backups and database synchronization
-- enterprise IT activities that happen infrequently and require a lot of
bandwidth for a short period of time.
In order to really verify that the system was functioning as expected,
we started a 100Mbit/s test stream across the service. Since our
initial service parameter included just 10 Mbit/s, we recorded 90Mbit/s
traffic loss. We then used the Bandwidth Calendaring web GUI to
increase our capacity to 100 Mbit/s for a duration of 10 minutes.
We clicked the go button and almost immediately the traffic
generator stopped reporting traffic loss.
This speedy reaction from the network, something that an enterprise
customer could immediately use, is a great way for service providers
to improve their service offerings and revenue streams.
There is another element to this tool that is worth noting. As the web
interface shows how this application will run in an international telecom
network, and not in confined test lab, we noted that the nodes were
geographically spread on a map and that latency was an element that
could be configured when making the bandwidth calendaring request.

Since we did not have an impairment generator in the lab, which would
have allowed us to simulate different link latencies, we could not verify
latency constraints would have worked, but we expect this evaluation
to be part of a follow-up test.

PARTICIPATING
COMPANY:

After we gave the optimize command we received a report that

confirmed that the tunnels we intended to split had indeed been split
and the ones that we wanted to merge were merged. We checked the
router configuration and verified that the changes had taken place.

Cisco WAN Automation Engine Tunnel

Split/Merge Manager
The final web application running over WAE that we verified is an
optimization tool called Tunnel Split/Merge.
The prime use case is a brownfield deployment where a large number
of Label Switched Paths (LSPs) already exist in the network. The idea
is that once WAE is introduced into the network, the service provider
could re-optimize the LSPs, removing those no longer needed and
optimizing others.
Another use case is where automatic bandwidth LSPs are used in the
network. In this instance, Cisco is trying to address the problem faced
when the bandwidth setup on the LSPs becomes so great that, in the
event of failure, the network is unable to signal for backup links.
Whatever the use case, being able to optimize the network and
essentially grab back wasted network resource is beneficial in
bandwidth-constrained networks.
To verify that the tool functions as it should, we specified maximum
and minimum bandwidth thresholds for all LSPs in the test
network. Cisco WAE queried the network and returned a list of
recommendations about which LSPs to split and which LSPs to merge.
We clicked Select all and then the Optimize button.

Figure 10: Ciscos WAN Automation Engine Tunnel Split/Merge Results

WAN Automation Engine Summary

Our tests of Ciscos WAN Automation Engine confirmed that the
functions Cisco claimed to support were indeed supported.
We recorded accurate link utilization reporting by the maintenance
tool and the bandwidth-on-demand tool. We also confirmed that
WAN Automation Engine can configure both Cisco and third-party
devices. This is an important aspect for CSPs as nearly all networks
are multi-vendor.
As we have seen, WAE enables the engineer to supervise the LSP
functions of a network at a higher degree of abstraction.
We also found, both as an independent test lab and subscribers to
a network, that the ease with which the bandwidth calendaring and

Figure 8: Ciscos WAN Automation Engine Bandwidth Calendaring

bandwidth-on-demand tools could be operated was very impressive

Application Interface
Cisco explained that latency could be taken from geographical
locations, based on latitude and longitude parameters, as well as by
using the Cisco Prime network management tool (which was not part
of the test) to monitor the network and actually add measurement data
to WAEs database.

and a great step forward.

With a software tool such as WAN Automation Engine and standardized
and defined protocols such as PCEP and REST, the network can be
automated and programmed dynamically and on request.
Figure 9: Ciscos WAN Automation Engine Tunnel Split/Merge
Application Interface

07

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

So now, what about the Network Service Orchestrator?

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

PARTICIPATING
COMPANY:

Cisco Network Service Orchestrator

enabled by Tail-f

Cisco Network Service Orchestrator

Service Provisioning

We first configured a service called Volvo that required a connection

Another important element in the management and orchestration

We started this part of the validation process with a view of the

portfolio Cisco provided to EANTC is the aforementioned Cisco

elements without any services configured. The NSO interface showed

-- we were not on the routers CLI, but on the NSO command prompt.

Network Service Orchestrator enabled by Tail-f.

five Customer Edge (CE) routers, connected to two provider edge routers
from Cisco and two Provider Edge routers from another major vendor.

EANTC is very familiar with the Network Service Orchestrator

In the core of the lab network we had an IP router. Our goal was to

(NSO): While we were validating its capabilities at Ciscos San Jose

configure an IP/MPLS Layer 3 Virtual Private Network (L3VPN) service

labs, some of the NSO team members were testing multi-vendor

on this multi-vendor, multi-operating system network. (The ability

interoperability in our Berlin-based lab. It is rare for EANTC to be

to manage multiple operating systems is a key attribute as even the

testing a product in parallel in two different labs.

Cisco provider edge routers were running slightly different operating

systems -- one router was running Cisco IOS, while the other had

So what is NSO? Cisco describes NSO as being akin to a translator

Cisco IOS-XR.)

software toolbox with two layers.

to cause any disaster, we applied the configurations.

This is really when the magic happened, because without much
effort, with just a small amount of information fed into NSO, it is now
possible to make new services come to life without having to type
hundreds of lines of code for the router. Instead, NSO has the ability to
create services in a higher language, allowing engineers to think about

the CE routers to send traffic between CE1 and CE2. Since the service
model included a 600Mbit/s shaper, we asked the Cisco team to

The second layer faces southbound, towards the network, and is called

increase the traffic rate to 700 Mbit/s and looked for dropped packets.

Network Elements Drivers (NEDs). This layer uses standards-based

We indeed lost 100 Mbit/s per direction of traffic, which proved to us

interfaces (such as the IETF NETCONF or SNMP) and is augmented by

that the bandwidth limiter part of the service model was working.

a CLI interface.
Figure 11: Network Topology as Discovered by Ciscos NSO

that is where NSO provides network service programmability. Between

the interfaces are two software modules called service manager and

There is not much magic initially in such tests. We had to manually

device manager. Using these two pieces of software, NSO has the

enable access on the routers for the software. We then synchronized

ability to understand service models and what is needed to implement

the NSO configuration database (CDB) with the device. We noted that

them on a device. Service models are defined in Yang (IETF RFC

Cisco was using two different interfaces to the devices: The Cisco

6020) which is, as the IETF RFC 6020 explains, a data modeling

devices used the CLI module, while NSO connected to the third-party

language used to model configuration and state data manipulated

router using NETCONF. Why the two approaches?

by the Network Configuration Protocol (NETCONF), NETCONF remote

The Cisco team said they wanted to demonstrate the practical
approach of both CLI and NETCONF in a way that a CSP would use
This represents a significant change in terms of the way services are

them in a live network today, adding that NETCONF could just have

created. It changes the focus from defining configuration to defining a

easily been used to connect the Cisco gear and CLI used to connect to

service, which NSO then configures on the relevant network devices.

the third-party router.

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

configuration on the device. Once we were certain we were not about

Cisco configuration. We then were able to use the tester attached to

program MPLS Label Switched Paths (LSPs) in the network.

08

the configuration we were sending to the device with the existing

By adding 15 lines of NSO configuration data, we counted 318 lines of

or other applications: In our earlier processes, WAE was using NSO to

So what can NSO do?

We then used an NSO command called commit dry-run to compare

quantify that during our validation process.

CLI, JSON-RPC, REST, and so on. This layer faces the network operator

procedure calls, and NETCONF notifications.

very similar to the configuration of a router, but with a big distinction

services instead of network element configuration and we were able to

One layer is a programmable interface to various service tools such as

However, it is between the two layers where the magic happens, as

from CE1 to PE1 and from CE2 connection to PE2. The activity looked

We now had a topology that included two CE routers and two PEs
routers, all from Cisco -- so we added two third-party routers to
the topology. This time the new service (one CE-to-PE connection)
required six lines of configuration in the NSO CLI interface and
resulted in 191 lines of device configuration and a functioning service.
The Cisco team explained that the devices were already discovered
simply by dragging the CE router onto a PE. Again, we sent tester
traffic to check that the new device was really added to the service.
The results of this initial test were impressive for a few reasons. First,
the NSO CLI we were using felt like a Cisco CLI, which was familiar and
comfortable. Thats because the NSO is able to mimic vendor CLIs:
If a network engineer is more comfortable with another vendors CLI,
NSO can adapt to behave like the other CLI, making it easier to use. As
a result, multi-vendor networks (once service and device models are
applied) could be operated using a single CLI.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

PARTICIPATING
COMPANY:

The fact that we required so few lines of configuration to create such

complex service (not only connectivity but also Quality of Service
configuration) also provided a real illustration of the benefits of this
software tool. The obvious next question was -- what happens when
we want to modify the service?

Cisco Network Service Orchestrator

Service Modification
Now that we had a fully functioning and tested IP/MPLS L3VPN
service running in the test network, it was time to modify some of the
configurations in the network.
Using the GUI interface we changed the AS number on one of the CE
routers (CE2) and checked that the configuration had taken place in the
device itself. Since traffic was actually running between all end points,
we could calculate that the service on this particular CE was interrupted
for a duration of 6 seconds and then came back to full operation.
We then used the GUI to change a CE connection to a different PE. All
we had to do was navigate to the end-point panel, choose a different
PE interface to apply to our chosen CE router and apply the change.
Again, we could verify the resulting configuration before approving and
then commit.
The interesting activity was not simply the change in device
configuration -- the whole test bed was sitting on a switch, so
changing a connection did not require re-cabling -- but also that NSO
cleaned up the configuration that was no longer needed on the initial
PE router.

Figure 12: Initial service configuration, all routers participating in the

service highlighted in purple by NSO

09

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

Figure 13: Resulting IP/MPLS L3VPN service (note CE4 re-homed by NSO)

Cisco Network Service Orchestrator

Service Restoration
Now let us assume that the network operations team is still not
convinced and that some of the network engineers are actually still
modifying router configurations and are not using Ciscos NSO. A
mechanism must exist in NSO to identify and hopefully painlessly
resolve such issues.
In order to check if NSO could actually deal with such a scenario in a
graceful manner, we asked one of the Cisco engineers to log onto one
of the CE routers and remove the BGP configuration. Once the engineer
typed in no router BGP, we immediately noted that all test traffic from

Figure 14: NSO Command Line Interface in re-deploy state

and to this CE router displayed loss.

When a situation like this occurs in the network, the root cause must
be identified and the issue resolved -- that is human nature!

Cisco Network Service Orchestrator

Service Restoration Due to Node Failure

We used the NSO CLI to compare the configuration we had in the

The next validation we explored was a situation in which a

network prior to the issue with the CE router. The NSO reported that a

configuration is ready to be applied but a device that needs to be

section of the configuration was missing. We used a command called

involved in the service has failed. This time we modified the shaper

re-deploy to override the configuration on the CE router. As the Cisco

bandwidth allowance from 600 Mbit/s to 1 Gbit/s. We used the dry-

team explained, NSO then compares the service model that should

run command again and were provided with a report highlighting the

have been on the device with the current configuration and applies

changes to be applied to the configuration. These are depicted in the

only the minimal missing configuration.

figure below.

We verified that the service was indeed back to normal operations and
moved on to the next test scenario.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

PARTICIPATING
COMPANY:

community. We then applied the configuration to the device group,

We also confirmed Ciscos statement that NSO is able to provision

providing the community string on the CLI, and watched as in a few

multiple devices, not all of them from Cisco. Configuring multi-vendor,

seconds, all nine routers were configured with SNMP read-only access

multi-generation operating systems is a novel way to deal with

and the correct community (public).

brownfield deployment as well as being very much in line with the drive
towards SDN. In fact, the creation of services through templates looked

This in an interesting and potentially useful concept. We asked Cisco

a lot more like programming the network than configuring the network.

which Element Management System (EMS) was responsible for the

devices in our validation network: In fact, NSO incorporates the device
management and that since it maintains connectivity with the devices
at all times, it can dynamically react to changes in the devices.

Step 3: The Cloud Services Creation and

Delivery Stage
Both Ciscos WAN Automation Engine and Network Service

Figure 15: NSO Commit dry-run output with future changes highlighted
Before applying the configuration, we shut down one of the routers
involved in the service. Now our commit command was rejected by NSO
on the basis that it could not connect one of the devices in the service.
Both service restoration tests showed that Ciscos NSO software is
able to react to changes in the network and save an operator from his
or her own mistakes. The software also demonstrated mechanisms
to allow the operator to easily control the activities that the software
intends to perform.

Cisco Network Service Orchestrator

Device Management

Cisco Network Service Orchestrator

Service Model Modification
Last, but certainly not least, we asked for access to the service model
used in the tests to see what would happen if we made changes to the
configuration.
After all, from the validation processes we had undertaken it was clear
that the service model is one of the key elements in the software,
which intrigued us. So we modified the QoS model, adding three
classes -- bronze, silver and gold -- and reloaded the packages.
Now when we tried to use the NSO CLI to apply QoS configuration
to our service, we received the three options that we had just added
to the service on all the devices (CE, PE and third-party devices).
We verified if the configuration was applied to the devices and were
no longer surprised to see that all devices now had the correct QoS
configuration modifications.

Obviously, if a CSP has a large network, adding all configuration

commands manually will be a rather tedious and long process and one
likely to introduce errors. The Cisco team explained that since NSO
understands both services and devices, an operator could create a
template for a configuration change and apply it to all devices.
We decided that validating that function would be a rather simple test
and asked the Cisco team to activate SNMP on all three device types
-- two Cisco IOS devices and a third-party device. Cisco created a
configuration that included ten commands in total (the third-party
configuration required four lines, while both Cisco IOS configurations
required three lines each). Each of the SNMP configuration commands
also included a variable, denoted by a dollar sign, with the SNMP

10

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

Orchestrator are software tools designed to enable a CSP to automate

and increase the speed of operating network devices as well as rolling
out new services.
As Cisco explained, both tools can be used to completely automate the
deployment of new services. The logic behind the next applicationsfocused validation process was that, according to Cisco, CSPs have
not been able to address the Small and Medium Business (SMB)
market effectively, since the cost of customer acquisition and rollout
in that market segment is prohibitive.
With greater control and flexibility, though, the market dynamics can
change, especially through the use of self-provisioning tools. Market
dynamics are not something that EANTC, as an independent test lab,
can comment on. What we can do, though, is test the ability of an
end-customer to provision a service and to use it without having to
interact at all with a CSP call center and without burdening a service
providers operations department. To achieve that, the whole service

Cisco Network Service Orchestrator (NSO)

Tests Summary
Our NSO-focused tests explored the full service life cycle, from service

life-cycle process -- from ordering through service delivery, service

management and quality assurance -- must be completely automated.

We were able to measure operational efficiency by showing that with

Cisco Cloud Managed VPN Solution for

Virtual Managed Services

minimal NSO configuration commands, we could create extensive

Cisco presented us with a system it created for one of its Tier 1 CSP

device configuration.

customers. The service was completely branded with said CSPs logos

creation, through modification, redeployment and device management.

and other corporate identity, which is why the figures we display in this
Of course, we cannot attest to the efforts involved in the creation

section are actually taken from dCloud: The same service exists as an

of the service and device models: That might be a good topic for a

evaluation scenario and was part of the dCloud verification.

follow-up validation process.

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

The scenario we were trying to build is a small business of two

locations that wants a VPN. We called the business (fondly) Lakshmis
Caf. The perspective of this process shifted from CSP to end user, the
owner of the caf. Acting as the owner, we were interested in connecting
both locations using a VPN as well as making sure that the free WiFi we
intend to provide in our imaginary caf was protected.

PARTICIPATING
COMPANY:

At this point Cisco explained that the speed is actually part of the

Our question to the vendor was simple: How will this work? Cisco

license. This meant that the virtual router that will be associated with

explained that once the service is ordered, its Web Security Virtual

the service will have a specific license associated with it.

Appliance (WSAv) will be instantiated in the CSP cloud and will

be responsible for performing the URL filtering function. With that
information in mind, we continued with the ordering process, while
we also informed Cisco that wed like to record the whole process
happening in the background once our order was completed.

The whole process of ordering the service was driven by a web

interface. We started by creating a new customer account for our caf.
As you can see in the figure below, there were several options on the
menu. We choose the Cloud VPN Full service since it included Firewall,
URL filter, Remote Access, and Cloud VPN with Internet Access.

Figure 18: Cisco Cloud VPN Speed Selection Panel

The next panel allowed us to choose the type of SSL license. This is
Figure 16: Cisco Cloud VPN Service Type Menu

Once we clicked to the next page we had to insert the number of

sites we wanted to connect and the number of users at each site. We
could also indicate the expected staff growth, which the Cisco team
explained was used for bandwidth calculation.

where could imagine that small and medium businesses would run
into difficulties. What is SSL VPN and why would I need it? That would

And a moment later, we were done! Once we moved on to the next

be the first question that would come to mind. In this case, since

panel we received a list of two devices that were going to cost us a

we did not know what SSL VPN was, we agreed with the most basic

lot of money. Since we had two sites and the devices also included

recommendation, using the logic, If you dont know what it is, you

WiFi Access Point function, we felt that our caf could benefit from

dont want to pay for it.

these elements and we moved on to the next panel. The latter was
an Amazon-like shipping address system that allowed us to send a

Cisco explained that the SSL VPN connection was actually part of the

router to each of our cafs. We then received a nice thank you for your

secure connection to the cloud and that it was required to secure the

purchase panel and, with this, the Cloud VPN service for Lakshmis

customer access to the service.

Caf was purchased.

Figure 17: Cisco Cloud VPN Company Profile Setting

Figure 19: Cisco Cloud VPN SSL License Panel

After this step was complete, we understood the point of some of the

This is really where the service began to take shape and the power

questions. The price for the basic service was displayed on the first
page. Based on the company profile, additional bandwidth could be
added, at cost, to the service. Cisco uses the company profile data to
recommend what they think will be appropriate for the customer size.
We decided to stick with the recommendations (remember, we are a caf
owner here, not network engineers) and moved on to the next page.

11

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

Figure 20: Cisco Cloud VPN URL Filtering Level Panel

Figure 21: Cisco Cloud VPN Device Panel

of virtualization came into play. The next panel allowed us to choose

URL filtering options. As a caf owner, we liked the idea that the folks
drinking at our establishment will not be able to visit malicious or
illegal websites, nor did we feel that our coffee shop should allow
access to adult material. We picked the medium-level URL filtering
service and turned our attention to Cisco.

Figure 22: Cisco Cloud VPN URL Filtering Shipping Address Panel

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

Cisco Cloud VPN: Behind the Scenes

Once the service was added, the next step was to wait for the routers
to be delivered. We skipped this phase, of course, and proceeded to the
part where the service actually spins up.
As Cisco clarified, once Lakshmis Caf receives the small routers,
the serial number of the routers will be manually inserted into the
same web interface panel. This will start the process of services
instantiation. We inserted the serial number to the web interface
and switched the view to the command line interface to monitor
what was happening.
The log scrolled for roughly 20 minutes until the system announced
that the service was ready to go. This included spinning up a Cisco
CSR1000v, a Web Security Virtual Appliance (WSAv) as well as a Cisco
ASA 1000V Cloud Firewall. As Cisco revealed, all virtual functions were
instantiated over Kernel Based Virtual Machines (KVM).
We then used Ciscos Network Services Orchestrator (NSO) to take
stock of the devices, virtual and otherwise, to verify that everything
we ordered was not only instantiated, but also manageable. Last, but

Table 1: Mapping to the ETSI NFV ISG Architecture

used for management (connected to NSO) and the second tunnel was
used for user data. The logic behind this is that Cisco assumes that
everything is over-the-top, which means NSO and other management
access will have to take place, possibly, over the Internet (hence the
management IPSec tunnel).
Last, but not least, we took two more actions: We added the second
caf to the service and at the end, deleted one of the sites from the
service. We performed both activities from the web interface and used
Ciscos NSO to verify both.

Production Environment Application

Demonstrations Project Squared and
Mobility IQ

ETSI NFV ISG Architecture

Element [1]

Cisco Cloud VPN Element

NFV Infrastructure (NFVI) a

collection of physical compute,
network and storage resources
over which virtual compute,
storage and network resources
are available

Cisco UCS hosting KVM

hypervisor

Virtualized Network Functions

(VNF) software instances of
similar or equal functionality to
physical functions

Virtual Router Cisco CSR1000v,

Virtual web security appliance
Cisco WSAV, Virtual Firewall
- Cisco ASA 1000V Cloud Firewall

Ciscos Reinhardt Quelle, operations architect of the companys

Virtualized Infrastructure
Manager (VIM)

OpenStack and OpenDayLight

that explains how Cisco is using OpenStack. As Reinhardt explained, the

VNF Manager

Cisco Elastic Service Controller

(ESC)

Orchestrator

The final piece of the puzzle was a hands-on demonstration -- but

not a test or validation -- of two software-as-a-service (SaaS)
applications that Cisco has developed for its customers. As Cisco
explained, both software tools were running in a live production
environment, which is why we could not get hands-on with them.

Intercloud (its cloud of clouds), explained that Cisco has been

committed to creating an open cloud environment for a while. He even
backed up his statements with a blog entry on the OpenStack web server
real magic, and the value proposition such applications have for a service

Cisco Network Services

Orchestrator enabled by Tail-F

Note 1: Based on ETSIs GS NFV 002 document.

not least, we verified there were two IPsec tunnels established using
the CLI of the two routers. Cisco explained that one IPsec tunnel was

PARTICIPATING
COMPANY:

provider, is that they actually just run -- service providers do not have to
spend capex on components to create the service.
As Reinhardt was walking us through the whole production system,
which included Ciscos own data centers as well as public cloud access
and CSP clouds, we noted that a node just came up in one of the
Interclouds service providers. It is no secret that Cisco signed several

From an end customer perspective, in this case the owner of Lakshmis

service providers around the world to be part of the Intercloud, as Light

Caf, we had a great experience. Without calling any support or

Reading reported several times during 2014. Nonetheless, seeing a node

ordering line, using a web interface only, we created a service that

come up, during a discussion on the system, was a nice coincidence.

included not only connectivity, but also services. Remaining with the
same perspective, we measured 20 minutes to get the whole service

As reported in September 2014, Deutsche Telekom (DT) has also

up and running, including all added functions (such as a firewall).

been part of the Intercloud ecosystem. Since EANTC is also a German

company (though clearly not the same size as Deutsche Telekom), we

From a service provider perspective, a customer created a rather

were interested in data governance. After all, in Germany, privacy laws

complex service without involving an operations team. Using the

are very different than in the US. For example, a service provider such

cloud, the various automation tools we already tested and open source

as DT has to be able to maintain control over the data used in any

software (OpenStack), no humans, other than the shipping department,

services and guarantee that the data remains in Germany. Reinhardt

were involved with the service rollout.

explained that data governance as well as encryption and security

are big aspects of Intercloud. We wanted to try and move data from

Cisco Cloud VPN Solution Life Cycle

Conclusion

Another observation is that rolling out additional services, with

Germany to other countries, but Cisco reminded us that the system is

more sites, would actually require the same amount of time, since

live and recommended that such tests should be done at a later date.

As Cisco explained, the whole backend system, which allows the

the instantiation of virtualized services should remain a constant --

service to be completely automated, is an instantiation of ETSIs NFV

they only differ from each other by licensing initially. This will be an

So instead of talking about theoretical aspects of Intercloud we

ISG architecture. If we try to map the elements we saw in this test, we

excellent point to validate in a follow-up test.

asked to move to the applications running on top of Intercloud.

see that the following elements map nicely to the NFV architecture:

12

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

Cisco reminded us that we already had our hands on one Intercloud-

PREVIOUS

TABLE OF CONTENTS

NEXT

LightReading
www.lightleading.com

REPORT

TESTING
PARTNER:

MARCH 2015

supported application -- dCloud. They added that the applications we

usage, and so on. Alongside these network-centric KPIs, Mobility IQ

were about to see were created by Cisco as a way for CSPs to further

offers answers to consumer questions such as Which floor is busier?

sell SaaS apps to their customers.

or Which bathroom is more packed? The theory with the latter

metrics is that an operator of a service within the WiFi area could

Cisco WebEx Squared

attract customers by sending discount notifications to the end-users.

Synergy Research places Ciscos WebEx as the global leader for SaaS

EANTC has been a contributing member of the ETSI Network Functions

Virtualization Industry Specification Group (NFV ISG) since its third
meeting in July 2013. The ISG has released three white papers
between its formation outlining the vision, drives, progress and results.
communicated that they were concerned to maintain momentum by

evolved over the years (adding functionality such as file sharing,

avoiding a protracted standards effort.

search and encryption), so did the resources they consume. For most
networked enterprises, collaboration tools are also mission-critical,

In this report we were able to verify that Ciscos virtualization solution

which puts the emphasis on resiliency and availability. All these

is real. As we showed in the report, Ciscos Cloud VPN appears to

were reasons for Cisco to work on an evolution of WebEx. The new

adhere to the NFV architecture. We also verified that the Cisco solution

collaboration tool is currently called Project Squared and is already

works with at least one third-party device and uses free software and

available on the web as well as in the iTunes Store and Google Play.

standardized interfaces, as defined by the IETF, to orchestrate and

manage services.

We installed Squared on our smartphones and also moved the

whole conversation to the web interface. Reinhardt explained that

This report shows real and tangible momentum, based on existing

because Squared is running over multiple clouds, when the recent Xen

standards and code: It just happens to come from the IP sectors

hypervisor bug hit (forcing RackSpace to reboot), the Squared service

giant. Chapeau.

was not affected at all.

Jambi I. Ganbar, Senior Technical Marketing Manager and Carsten

We can confirm that Project Squared is real and so can you, by simply
Cisco, based on the web-based management system they showed us, has

Conclusion: Meaningful Developments

As the concepts of NFV started taking flight, the ISGs CSP members

web conferencing, with 51% market share. As collaboration tools

downloading the applications and using them. We can also confirm that

PARTICIPATING
COMPANY:

Rossenhvel, Managing Director, European Advanced Networking

Figure 23: Mobility IQ Zone Movement example

an extensive infrastructure spanning a large number of clouds -- public,

CSP-based and their own. We certainly hope that in the next phase of

Cisco demonstrated an extensive and slick-looking web interface

testing we could spend more time and delve into this infrastructure.

that provided an overview of the various KPIs and allowed the CSP

Test Center AG (EANTC) (http://www.eantc.de/), an independent test

lab in Berlin. EANTC offers vendor-neutral network test facilities for
manufacturers, service providers, and enterprises.

to further tailor the view and provide access to an organization level.

Mobility IQ

The interesting aspect for us, other than the details of how this works,

Cisco has just released Mobility IQ. This SaaS application is designed

was the claim that Cisco made about the way it hopes the service will

to offer deep visibility into small cell and WiFi access. The app is

be used by CSPs: Cisco intends to sell Mobility IQ to service providers

targeted at CSP customers that, in turn, are supposed to offer this

and perform the whole integration, while service providers will only

service to their customers.

need to re-brand the service.

And what does the service bring? Well, Cisco was careful to pick

Again, since the system was live, we could not gain access to such

a customer it said was live and an operator of a major sports and

elements as the statistics collection or actually simulate WiFi access

concert hall in a large metropolitan area. The tool provided statistics

point failure. We can confirm that the system appears to be real and

about technical key performance indicators (KPI) such as WiFi

operational and we hope, as with Project Squared, to see the system in

attachment rate, access point health, bandwidth usage, spectrum

action during a future test.

13

Validating Ciscos Service Provider Virtualization & Cloud Portfolio

PREVIOUS

TABLE OF CONTENTS

NEXT