Vxlan Design Guide

White Paper
Cisco IP Fabric for Media

Design Guide
April 2018
Rahul Parameswaran (rparames)

CISCO SYSTEMS, INC.
© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 37
Contents
Introduction .............................................................................................................................................................. 3
Frame accurate switching ....................................................................................................................................... 4
Nexus 9000 for IP fabric for Media ......................................................................................................................... 4
Spine and leaf design with network controller ...................................................................................................... 5
Why use a layer 3 spine and leaf design .............................................................................................................. 5
Non-blocking multicast in CLOS architecture ........................................................................................................ 5
Why Media Fabrics can’t rely on ECMP and PIM alone ........................................................................................ 6
Flow orchestration with Cisco DCNM media controller ......................................................................................... 6
Cisco DCNM media controller features ................................................................................................................. 6
Cisco DCNM media controller installation ............................................................................................................. 7
Fabric topology ..................................................................................................................................................... 7
Fabric configuration using Power-On Auto Provisioning (POAP) .......................................................................... 9
Fabric configuration using the command-line interface ....................................................................................... 12
Topology discovery ............................................................................................................................................. 14
Host discovery .................................................................................................................................................... 15
Host policy .......................................................................................................................................................... 16
Flow policy .......................................................................................................................................................... 17
Flow setup........................................................................................................................................................... 18
Flow visibility and bandwidth tracking ................................................................................................................. 18
Flow statistics and analysis ................................................................................................................................. 20
Flow alias ............................................................................................................................................................ 20
Events and notification ........................................................................................................................................ 21
Precision Time Protocol ...................................................................................................................................... 22
Quality of Service (QoS) ..................................................................................................................................... 23
API integration with the broadcast controller ....................................................................................................... 23
Failure handling .................................................................................................................................................. 24
Cisco DCNM media controller connectivity options: fabric control ...................................................................... 26
Designing the control network ............................................................................................................................. 28
Cisco IP fabric for Media architecture ................................................................................................................. 29
Multi-Site and PIM Border ................................................................................................................................... 31
Guidelines and Limitations ................................................................................................................................... 32
Topology discovery ............................................................................................................................................. 32
Flow setup........................................................................................................................................................... 32
Host policy .......................................................................................................................................................... 32
Flow policy .......................................................................................................................................................... 33
Design ................................................................................................................................................................. 33
Live and File based workflows on same fabric .................................................................................................... 33
Single switch with controller ................................................................................................................................ 34
Single switch without a controller ........................................................................................................................ 35
Conclusion ............................................................................................................................................................. 36
For more information............................................................................................................................................. 36
Introduction
Today, the broadcast industry uses an SDI router and SDI cables to transport video and audio signals. The SDI
cables can carry only a single unidirectional signal. As a result, a large number of cables, frequently stretched over
long distances, are required, making it difficult and time-consuming to expand or change an SDI-based
infrastructure.
Cisco IP Fabric for Media helps you migrate from an SDI router to an IP-based infrastructure (Figure 1). In an IP-
based infrastructure, a single cable has the capacity to carry multiple bidirectional traffic flows and can support
different flow sizes without requiring changes to the physical infrastructure.
An IP-based infrastructure with Cisco Nexus 9000 Series switches:
● Supports various types and sizes of broadcasting equipment endpoints with port speeds up to 100 Gbps
● Supports the latest video technologies, including 4K and 8K ultra HD
● Allows for a deterministic network with zero packet loss, ultra low latency, and minimal jitter, and
● Supports the AES67 and SMPTE-2059-2 PTP profiles
Figure 1(a) SDI router
Figure 1(b) IP fabric
The Society of Motion Picture and Television Engineers (SMPTE) 2022-6 standard define the way that SDI is
encapsulated in an IP frame and SMPTE 2110 defines how video, audio and ancillary data is carried over IP.
Similarly, Audio Engineering Society (AES) 67 defines the way that audio is carried over IP. All these flows are
typically User Datagram Protocol (UDP) and IP multicast flows. A network built to carry these flows must help
provide zero-drop transport with guaranteed forwarding, low latency, and minimal jitter.
There are multiple design options that are available today based on a given use case
1. Spine and Leaf CLOS based architecture using Nexus 9000 series switches with Data Center Network
Manager (DCNM) Controller – Provides a flexible and scalable architecture that is suitable for studio
deployments.
2. A single switch, Nexus 9000 series with DCNM controller – Provides a design that is identical to an SDI router.
Simple architecture with DCNM controller offering security and flow visibility. Studio or OBVAN type
deployment.
3. A single switch, Nexus 9000 without a controller – Provides a design that is identical to an SDI router. Simple
architecture. Switch operates like any multicast router.
Frame accurate switching

When switching from one source to another, it is important to ensure the switch occurs at a frame boundary. The
ways to ensure frame accurate switching are as follows:
Network Timed Switching: Here the network is responsible to switch on a frame boundary. This is not possible in a
standard off the shelf switch as it requires FPGA to switch at the frame boundary.
Source Timed Switching: Source timed switching requires the source to modify the UDP port number when a
switch is triggered. At a precise time interval, the source is instructed to change the destination UDP port number
of the signal. The network switch is programmed with a rule to steer the signal to the receiver just before the
source switches their destination UDP port. While this works, it does not scale when the network involves multiple
devices (routers and switches) and limits the ability to switch flows across different sites.
Destination Timed Switching: The destination subscribes or joins the new flow before leaving the old flow and
switches at the frame boundary. Destination timed switching does not require special capabilities in the network
and can also scale across multiple network switches and sites.
Given the goal of the industry is to use commercial off-the-shelf switches, in general it has chosen to move forward
with destination timed switching and Cisco has adopted that model.
Nexus 9000 for IP fabric for Media

Nexus 9000 Series delivers proven high performance and density, low latency, and exceptional power efficiency in
a range of form factors. It also performs line rate multicast replication with minimal jitter. Each switch can operate
as a PTP boundary clock and supports SMPTE 2059-2 and AES67 profile.
Listed below are supported Nexus 9000 switches with their role
Table 1. Nexus 9000 switch port capacity and supported role
Part Number Description Supported Mode

N9K-C92160-YC 48x10/25G+4x100G(6x40G) Leaf with Controller/Standalone Switch
with/without Controller
N9K-C93180YC-EX and FX 48x10G/25G+ 6x100G Leaf with Controller/Standalone Switch

N9K-C93108TC-EX and FX 48x1G/10G Base T + 6x100G Leaf with Controller/Standalone Switch
N9K-C9236C 36x100G/40G (Supports 4x10G,4x25G) Leaf or Spine with Controller/Standalone
Switch with/without Controller
N9K-C9272Q 72x40G Port (Supports 4x10G on ports 36-71) Leaf or Spine with Controller/Standalone
Switch with/without Controller
Part Number Description Supported Mode
9500 with N9K-X9636C-R 36x100G Spine with Controller/Standalone Switch
9500 with N9K-X9636Q-R 36x40G Spine with Controller/Standalone Switch
Spine and leaf design with network controller

Figure 2. Multiple-spine and multiple-leaf architecture with Cisco DCNM media controller
Why use a layer 3 spine and leaf design

Spine and leaf CLOS architecture has proven to be flexible and scalable and is widely deployed in modern data
center design. No matter where the receiver is connected, the path always involves a single hop through the spine,
thereby essentially guaranteeing deterministic latency.
Although a Layer 2 network design may seem simple, it has a very large failure domain. A misbehaving endpoint
could potentially storm the network with traffic that is propagated to all devices in the Layer 2 domain. Also, in a
Layer 2 network, traffic is always flooded to the multicast router or querier, which can cause excessive traffic to be
sent to the router or querier, even when there are no active receivers. This results in non-optimal and non-
deterministic use of bandwidth.
Layer 3 multicast networks contain the fault domain and forward traffic across the network only when there are
active receivers, thereby ensures optimal use of bandwidth. This also provides granular application of filtering
policy that can be applied to a specific port instead of all devices like in case of a layer 2 domain.
Non-blocking multicast in CLOS architecture

SDI routers are non-blocking in nature. A single Ethernet switch such as the Nexus 9000/9500 is also non-
blocking. A CLOS architecture provides flexibility and scalability, however, there are a few design considerations
that needs to be taken into consideration to ensure a CLOS architecture remains non-blocking.
In an ideal scenario, the sender leaf (First hop router) sends one copy of the flow to one of the SPINE switch. The
SPINE creates “N” copies, one for each receiver LEAF switch that has interested receivers for that flow. The
receiver leaf (Last hop router) creates “N” copies of the flow, one per local receiver connected on the leaf. At times,
especially when the system is at its peak capacity, you could encounter a scenario where sender leaf has
replicated a flow to a certain spine, but the receiver leaf cannot get traffic from that spine as its link bandwidth to
that spine is completely occupied by other flows. When this happens, the sender leaf must replicate the flow to
another spine. This results in the sender leaf using twice the bandwidth for a single flow.
To ensure the CLOS network remains non-blocking, a sender leaf must have enough bandwidth to replicate all of
its local senders to all spines. By following this guideline, the CLOS network can be non-blocking.
Bandwidth of all senders connected to a leaf must be equal to the bandwidth of the links going from that
leaf to each of the spine. Bandwidth of all receivers connected to a leaf must be equal to the aggregate
bandwidth of all links going to all spines from that leaf.
For example: A two spine design using N9k-C93180YC-EX, with 6x100G uplinks, 300G going to each spine
can support 300G of senders and 600G of receiver connected to the leaf.
In a broadcasting facility, most of the endpoints are unidirectional – camera, microphone, multiviewers etc. Also,
there are more number of receivers than senders (ratio is 4:1). Also, when a receiver no longer needs a flow, it
leaves the flows, freeing up the bandwidth. Hence, the network can be designed with placement of senders and
receivers such that the CLOS architecture becomes non-blocking.
Why Media Fabrics can’t rely on ECMP and PIM alone

In IT data centers, Equal-Cost Multipath (ECMP) is extremely efficient because most traffic is TCP based, with
millions of flows. A Cisco IP Fabric for Media network has fewer flows, but the flows use more bandwidth, and all
are UDP flows. In this scenario, ECMP may not always be efficient, resulting in hash collusion. Hence, an IP
network needs a controller to orchestrate flows based on bandwidth availability.
Flow orchestration with Cisco DCNM media controller

The Cisco Data Center Network Manager (DCNM) media controller provides a variety of features and functions.
One important function is to ensure that flow is set up between sources and receivers with guaranteed bandwidth
available for the flow. Every flow request sent either by an IGMP join message from a receiver or an API call from
the broadcast controller triggers the media controller to compute an available path in the network and to program
the path end to end. This process helps ensure efficient use of bandwidth. It also helps prevent hash collusion,
which could potentially occur with ECMP.
Cisco DCNM media controller features

The DCNM media controller is an extension to Cisco DCNM. It includes all the features available with DCNM plus
the following media controller features:
● Fabric configuration using Power-On Auto Provisioning (POAP) to help automate configuration
● Topology and host discovery to dynamically discover the topology and host connection
● Flow orchestration to help set up traffic flow with guaranteed bandwidth
● Flow policies, including bandwidth reservation and flow policing
● Host policies to secure the fabric by restricting senders and receivers
● Flow visibility and analytics, including end-to-end path visibility with bit rate set on a per-flow basis
● Events and notifications for operation and monitoring
● Northbound API integration with the broadcast controller
For more information about DCNM, see https://www.cisco.com/c/en/us/support/cloud-systems-management/data-
center-network-manager-10/model.html.
Cisco DCNM media controller installation

For the steps to install the DCNM media controller, see https://www.cisco.com/c/en/us/support/cloud-systems-
management/prime-data-center-network-manager/products-installation-guides-list.html.
The recommended approach is to set up the DCNM media controller in native high-availability mode.
After DCNM is installed, you must manually configure it to work in media-controller mode. You accomplish this by
connecting to the DCNM server through Secure Shell (SSH) and logging in with the username root.
[root@mc ~]# appmgr stop dcnm
[root@mc ~]# appmgr set-mode media-controller
With OVA installation, ensure DCNM is deployed using Large Configuration (four vCPUs and 12G RAM) and in
Thick Provisioned mode.
Fabric topology
The number and type of leaf and spine switches required in your IP fabric depend on the number and type of
endpoints in your broadcasting center.
Follow these steps to help determine the number of leaf switches you need:
Count the number of endpoints (cameras, microphones, gateway, production switchers etc.) in your broadcasting
center. For example, assume that your requirements are as follows:
● Number of 40-Gbps ports required: 40

The uplink bandwidth from a leaf switch to a spine switch must be equal to or greater than the bandwidth
provisioned to endpoints. Tables 1 and 2 list the supported switches and their capacities. Figure 3 shows the
network topology.
Table 2. Supported leaf switch
Leaf Switch Endpoint Capacity Uplink Capacity

Cisco Nexus 9236C Switch 25 x 40-Gbps endpoints 10 x 100-Gbps (1000-Gbps) uplinks
Cisco Nexus 9272Q Switch 36 x 40-Gbps endpoints 36 x 40-Gbps (1440-Gbps) uplinks

Cisco Nexus 92160YC-X Switch 40 x 10-Gbps endpoints 4 x 100-Gbps (400-Gbps) uplinks
Cisco Nexus 93180YC-EX Switch 48 x 10-Gbps endpoints 6 x 100-Gbps (600-Gbps) uplinks
Cisco Nexus 93108TC-EX Switch 48 x 10-Gbps endpoints 6 x 100-Gbps (600-Gbps) uplinks
Table 3. Supported spine switch
Spine Switch Number of Ports

Cisco Nexus 9236C Switch 36 x 100-Gbps ports
Cisco Nexus 9272Q Switch 72 x 40-Gbps ports
Cisco Nexus 9508 with N9K-X9732C-EX Line Card 256 x 40-Gbps ports
Cisco Nexus 9508 with N9K-X9636Q-R Line Card 288x 40-Gbps ports
Cisco 9508 with N9K-X9636C-R Line Card 288x 100-Gbps ports
● The 9236C can be used as a leaf switch for 40-Gbps endpoints. Each supports up to 25 x 40-Gbps
endpoints and requires 10 x 100-Gbps uplinks.
● The 93180YC-EX can be used as a leaf switch for 10-Gbps endpoints. Each supports up to 48 x 10-Gbps
endpoints and requires 6 x 100-Gbps uplinks.
● The 93108TC-EX can be used as a leaf switch for 1/10GBASE-T endpoints. Each supports up to 48 x
1/10GBASE-T endpoints with 6 x 100-Gbps uplinks.
● 40 x 40-Gbps endpoints would require 2 x 9236C leaf switches with 20 x 100-Gbps uplinks.
● 160 x 10-Gbps endpoints would require 4 x 93180-EX leaf switches with 24 x 100-Gbps uplinks.
● 70 x 1-Gbps endpoints would require 2 x 93108-EX leaf switches with 4 x 100-Gbps uplinks. (Not all uplinks
are used.)
● The total number of uplinks required is 48 x 100 Gbps.
● The 9500 with N9K-X9636C-R line card or a 9236C can be used as a SPINE.
● With 9236C, each switch supports up to 36 x 100-Gbps ports. Two spine switches with 24 x 100-Gbps ports
per spine can be used.
● With 9508 and N9K-X9636C-R line card, each line card supports 36x100G ports. Two line cards with a
single spine switch can be used.
Figure 3. Network topology with 9236c as spine
Figure 4. Network topology with 9508 as spine
Fabric configuration using Power-On Auto Provisioning (POAP)

POAP automates the process of upgrading software images and installing configuration files on Cisco Nexus
switches that are being deployed in the network. When a Cisco Nexus switch with the POAP feature boots and
does not find the startup configuration, the switch enters POAP mode, locates the DCNM Dynamic Host
Configuration Protocol (DHCP) server, and bootstraps itself with its interface IP address, gateway, and DCNM
Domain Name System (DNS) server IP addresses. It also obtains the IP address of the DCNM server to download
the configuration script that is run on the switch to download and install the appropriate software image and device
configuration file (Figure 5).
Figure 5. POAP process
The DCNM controller ships with configuration templates: the Professional Media Network (PMN) fabric spine
template and the PMN fabric leaf template. The POAP definition can be generated using these templates as the
baseline. Alternatively, you can generate a startup configuration for a switch and use it during POAP definition.
(The section “Fabric Configuration Using the Command-Line Interface” later in this document provides steps for
creating a configuration manually.)
When using POAP, you follow these steps:
● Create a DCHP scope for temporary IP assignment.

● Upload the switch image to the DCNM image repository.
● Generate the switch configuration using startup configuration or a template.
Figures 6 through 11 illustrate these steps.
Figure 6. DCNM > Configure > POAP
Figure 7. DCNM > Configure > POAP > DHCP Scope
Figure 8. DCNM > Configure > POAP > Images and Configurations
Figure 9. DCNM > Configure > POAP > POAP Definitions
Figure 10. DCNM > Configure > POAP > POAP Definitions
Figure 11. Generate configuration using a template: DCNM > Configure > POAP > POAP Definitions > POAP Wizard
Fabric configuration using the command-line interface

Fabric setup can be performed through the switch Command-Line Interface (CLI). The main components include
an Interior Gateway Protocol (IGP) such as Open Shortest Path First (OSPF) for unicast routing and PIM for
multicast routing. However, PIM is set up in passive mode because the DCNM media controller provisions the PIM
Outgoing Interface (OIF)) and does not use protocol negotiation. Using Nonblocking Multicast (NBM) mode in the
media controller, the switch is informed of the DCNM controller IP address and credentials. TCAM carving is used
for host security policies and flow statistics.
The configurations are shown here.
Setting up the NBM mode controller

! Enable the feature
feature nbm
feature nxapi
! Enable mode controller and specify controller IP

nbm mode controller
controller ip <dcnm_vip_address> vrf <vrf_name>
controller-credentials username admin password 0 <password>
! Verification
show nbm controller
VRF IP Role Status Online-

management 10.23.234.120 ACTIVE ONLINE 2016-12-04
14:43:00.151088
Configuring TCAM carving
! RACL TCAM for Host Policy and ing-l3-vlan-qos for flow statistics
! Requires a switch reload post configuration
! Only required on LEAF switches
hardware access-list tcam region ing-racl 512

hardware access-list tcam region ing-l3-vlan-qos 256
hardware access-list tcam region ing-nbm 1536
Configuring PIM and IGP (OSPF)

! PIM is enabled but in passive mode, IGP needed for any unicast traffic on
fabric
feature pim
feature ospf
router ospf 1
maximum-paths 36
! Maximum paths must be greater than number of uplinks between a single leaf and
spine layer. Default is 8 which is sufficient in most scenarios
interface etx/y
ip pim sparse-mode
ip pim passive
ip router ospf 1 area 0
Configuring the interfaces

! Fabric Port Configuration (links between leaf and spine)
interface Ethernet1/50
no switchport
ip address x.x.x.x/x
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
ip pim passive
no shutdown
! Host Port Configuration (link between switch and end host)

ip ospf passive-interface
ip pim sparse-mode
ip pim passive
ip igmp version 3
ip igmp immediate-leave
ip igmp suppress v3-gsq
no shutdown
! Host Port Configuration when using an SVI with port as a trunk or access port
interface Ethernet 1/2

switchport mode access
switchport access vlan 100
spanning-tree port type edge
interface vlan 100

ip pim sparse-mode
ip pim passive
vlan configuration 100

ip igmp snooping fast-leave
Topology discovery
The DCNM media controller automatically discovers the topology when fabric is brought up using POAP. If the
fabric is provisioned through the CLI, the switches need to be manually discovered by DCNM. Be sure that fabric
has been discovered after you set the default flow and host policy and before any endpoints are connected and
discovered on the fabric. Figures 12 and 13 show the steps required to discover the fabric.
Figure 12. DCNM > Inventory > Discover Switches > LAN Switches
Figure 13. DCNM > Media Controller > Topology
Host discovery
A host can be discovered in the following ways:
● When the host sends an Address Resolution Protocol (ARP) request for its default gateway: the switch
● When the sender host sends a multicast flow
● When a receiver host sends an IGMP join message
● When the host is manually added through an API
You can find host information by choosing Media Controller > Host.
The host is populated on the topology page, providing a visual representation showing where the host is
connected.
Only hosts that are discovered are displayed on the topology page.
Figure 14(a) DCNM > Media Controller > Topology
Figure 14(b) DCNM > Media Controller > Hosts
Host policy
Host policy is used to prevent or allow a sender to send traffic to certain multicast groups and a receiver to
subscribe to certain multicast groups. Default policy can use a whitelist or a blacklist model. Sender host policy is
implemented with an ingress access-list programmed on the sender leaf. Receiver host policy is implemented by
applying an IGMP filter on the receiver leaf.
The host policy page includes all policies created on DCNM through the GUI or API. It also indicates the devices to
which a given policy is applied. To see this information, click the “i” on the host policy page. (Figure 15).
Configuring the default host policy

Be sure to make any changes to the default host policy before any active host appears on the network. The default
host policy specifies whether to use a blacklist (allow all by default) or a whitelist (deny all by default) and should be
defined before hosts are added to the network.
Changes can be made to a host-specific policy at any time. However, no changes can be made to the default
policy anytime it is applied to any host. In order to make changes to the default policy, ports facing the hosts must
be shut down, the flows then time out and DCNM controller removes the policy associations on the switch.
Figure 15. DCNM > Media Controller > Policies > Host Policies
Flow policy
The flow policy is used to specify the flow properties, including the flow bandwidth and Differentiated Services
Code Point (DSCP) marking. The default policy is applied to flows that do not have a specific policy defined. The
flow policy helps ensure that sufficient bandwidth is provisioned on the network. It also helps ensure that the
sender does not send traffic at a rate higher than the rate provisioned using the policy. Excessive traffic is policed.
(Figure 16).
Configuring the default flow policy

Changes to the default flow policy must be made before any active flows exist in the system. The controller ships
with a default flow size of 3 Gbps and a DSCP of CS1.
Figure 16. DCNM > Media Controller > Policies > Flow Policies
Flow setup
Flow setup can be accomplished in two ways. A receiver, using Internet Group Management Protocol (IGMP) can
request for a flow, or the broadcast controller can setup a flow using an API. Details on the API and the API
integration is discussed in API Integration with the Broadcast Controller section.
Flow visibility and bandwidth tracking

One of the broadcast industry’s biggest concerns in moving to IP is maintaining the capability to track the flow path.
The DCNM media controller provides end-to-end flow visibility on a per-flow basis. The flow information can be
queried from the DCNM GUI or through an API.
One can view bandwidth utilization per link through the GUI or an API. Also, when the Bandwidth check box is
selected, the color of the links change based on their utilization. See Figures 17, 18, and 19.
Figure 17. DCNM > Media Controller > Topology > Multicast Group
Figure 18. DCNM > Media Controller > Topology and Double-Click Link
Figure 19. DCNM > Media Controller > Topology
Flow statistics and analysis

DCNM maintains a real-time per-flow rate monitor. It can provide the bit rate of every flow in the system. If a flow
exceeds the rate defined in the flow policy, the flow is policed, and the policed rate is also displayed. Flow
information is cached for a period of 24 hours. Flow information can be exported and stored for offline analysis
(Figure 20).
Figure 20. DCNM > Media Controller > Flow Status
Flow alias
Every flow on the fabric is either a video, audio or ancillary flow. For an operator to recognize a flow a more
meaningful name can now be associated with a multicast group. Flow Alias provides option to name a flow. The
flow alias name can be referenced throughout the DCNM GUI instead of the multicast group. (Figure 21).
Figure 21. DCNM > Media Controller > Flow Alias
Figure 22. DCNM Flow View with Flow Alias
Events and notification

The DCNM media controller logs events that can be subscribed to using Advanced Message Queuing Protocol
(AMQP). The events are also logged and can be viewed through the GUI. Every activity that occurs is logged: a
new sender coming online, a link failure, a switch reload, a new host policy pushed out, etc. (Figure 23).
Figure 23. DCNM > Media Controller > Events
Precision Time Protocol

Precision Time Protocol (PTP) is used to provide clock information to all endpoints to help ensure that they are
time synchronized. If you need PTP, you must enable the boundary clock function on all switches. The fabric does
not support multicast routing for PTP packets.
The active and passive grandmaster switches are connected to the leaf switch.
Two profiles are supported: 2059-2 and AES 67. Any profile can be enabled on any interface. The slave switches
connected to the grandmaster switch must be configured with the same PTP parameters (Figure 24).
Example
feature ptp
! ptp source IP can be any IP. If switch has a loopback, use the loopback IP as
PTP source
ptp source 1.1.1.1
ptp
ptp delay-request minimum interval smpte-2059-2 -2
ptp announce interval smpte-2059-2 0
ptp sync interval smpte-2059-2 -3
Figure 24. Grandmaster and passive clock connectivity
Quality of Service (QoS)

QoS provides differential treatment to traffic based on its DSCP markings. The suggested markings for traffic
traversing the fabric is as indicated in the table below (Table 4).
Table 4. QoS
Service Class (Flow/Control) DSCP Values Queue
PTP Time and Sync EF Control/Priority Level 1

Intercom Audio EF Priority Level 1
AES67, 2110 Audio AF41 Priority Level 2 or Bandwidth
2022-6, 2110 Video AF41 Priority Level 2 or Bandwidth

2110 Ancillary CS3 Bandwidth
Other/Unicast 0 Best Effort/Default
API integration with the broadcast controller

The broadcast controller integrates with the DCNM media controller through open REST APIs. The integration
helps ensure that the end operator experience remains unchanged while the solution moves from an SDI network
to an IP network. The broadcast controller remains synchronized with the DCNM network controller (Figure 25).
Figure 25. Integration of broadcast and DCNM media controllers
For a list of APIs, see https://developer.cisco.com/site/data-center-network-manager/#overview.
Failure handling
Most deployments implement redundancy by provisioning parallel networks and connecting endpoints to both
networks (for example, 2022-7). Any failure on one network will not affect traffic to endpoints because a copy of the
frame is always available on the redundant network (Figure 26).
Within a fabric, when a failure takes place, such as a link down or a spine failure, recovery mechanisms built into
the DCNM media controller will help ensure that traffic converges. However, a few seconds may be needed to
reprogram the flows on other available paths. Flows are not moved back to the original path after the original path
recovers. New flows use the recovered path. The DCNM media controller provides a list of flows affected by any
failure.
Figure 26. Redundant IP fabrics: Completely independent of each other
The following types of failures may occur:
● Link flap between a leaf and spine (fabric link)

● Spine reload or power down
● Leaf reload or power down
● Loss of connectivity between switch and DCNM media controller
● DCNM failure (server reload or power down)
Link failure
When a link fails between a leaf and spine, flows are moved to other available links. In scenarios where available
bandwidth is not sufficient to move all affected flows, the media controller will move certain flows (random). For the
flows that it was not able to service, an event will be generated notifying the user about the affected flows.
When bandwidth becomes available in the future, the flows will be programmed after an IGMP join message is
received from a receiver.
Spine failure
A spine reload causes all flows routed on that spine to fail. The media controller moves the affected flows to other
spine switches. In scenarios where available bandwidth is not sufficient to move all affected flows, the media
controller will move certain flows (random). For the flows that it was not able to service, an event will be generated
notifying the user about the affected flows.
When bandwidth becomes available in the future, the flows will be programmed after an IGMP join message is
received from a receiver.
Leaf failure
A Leaf failure will result in the loss of traffic to endpoints connected to that leaf.
After recovery, the endpoints will have to resend IGMP join messages.
Connectivity loss between Cisco DCNM and switch
The DCNM media controller is the control plane for the fabric. When connectivity is lost between the switch and the
controller, no new control plane activities are possible. New flows cannot be programmed, and flows cannot be
removed. Existing flows will continue to work. The switch will periodically retry to connect to DCNM.
Cisco DCNM failure

DCNM high availability is implemented using an active and cold standby pair. The databases of the active and
standby instances are synchronized. When the active instance fails, the standby instance detects the failure and
assumes the active role.
DCNM high-availability failover takes about three minutes to complete, during which time no new control-plane
activities can be performed (Future release will provide active, hot standby HA implementation).
Cisco DCNM media controller connectivity options: fabric control

The switches in the IP fabric can communicate with DCNM using Out-Of-Band (OOB) management, OOB front-
panel ports, or in-band management.
DCNM ships with two Network Interface Cards (NICs): Eth0 and Eth1. POAP works on Eth1, with the DHCP server
on DCNM configured to listen to DHCP messages on Eth1 only. Switches can use POAP using the management
port or a front-panel port. For POAP to work, the OOB management port or the front-panel port used for
communication with DCNM must be on the same VLAN as the DCNM Eth1 port.
Figures 27, 28, and 29 show the control options.
Figure 27. Using OOB management ports to communicate with Cisco DCNM media controller
Figure 28. Using OOB Front-Panel Ports to Communicate with Cisco DCNM media controller
Figure 29. Using Inband Ports to Communicate with Cisco DCNM media controller
The following CLI snippet shows the configuration using the OOB management port:
interface mgmt0
vrf member management
ip address 1.2.3.4/24
nbm mode controller
controller ip <ip address> vrf management
controller-credentials username admin password 7 ljw39655
The following CLI snippet shows the configuration using the OOB front-panel port:
! Define Fabric Control VRF
vrf context fabric_control
! Specify the VRF name under NBM controller configuration

nbm mode controller
controller ip <dcnm_ip> vrf fabric_control
description to_fabric_control_network
vrf member fabric_control
no shutdown
CLI snippet when using Inband

! Create a loopback interface on each switch
! Advertise the loopback into IGP
! Extend the IGP to DCNM Control Network
interface loopback 100

ip address 1.2.3.4/24
ip router ospf area 0
nbm mode controller

controller ip <ip address> vrf default
controller-credentials username admin password 7 ljw39655
Designing the control network

The control network can be divided into two segments. One is fabric control network, which includes the network
between the IP fabric and the DCNM media controller. The other is the endpoint control network, which enables the
broadcast controller to communicate with the endpoints and the DCNM media controller.
Figure 30 shows the logical network connectivity between the broadcast controller, endpoints, and DCNM.
The control network typically carries unicast control traffic between controllers and endpoints.
Figure 30. Control network
Cisco IP fabric for Media architecture

Figures 29 through 31 shows various deployment topologies using Cisco IP Fabric for Media.
Figure 29 describes a topology that supports a redundant IP network that can carry 2022-6/7 or 2110 flows.
Endpoints must be capable of supporting hitless merge.
Figure 30 shows the flexibility offered by a SPINE and LEAF design that can be used in a distributed studio
deployment model.
Figure 31 shows a possible deployment scenario to connect a remote site using the concept of a remote leaf. The
leaf is not configured any different from a leaf in the main facility. It is a part of the fabric.
Figure 31. Redundant IP networks
Figure 32. Distributed studios
Figure 33. Remote leaf
Multi-Site and PIM Border

To aid remote production and to interconnect various fabrics with each other or to connect a media fabric to an
external PIM network, the multi-site/PIM border feature has been introduced. This solution uses PIM SSM (source
specific multicast) to enable flow setup between remote senders and receivers. For a receiver to request flow from
a remote sender, it must send the IGMPV3 report, along with the sender and group address. Using the unicast
routing table, DCNM and the switch creates and propagates PIM (S,G) join towards the source. When the fabric
that receives the PIM (S,G) join, it programs the flows to go all the way from the sender to the receiver. DCNM
intelligently manages the inter site link bandwidth, ensuring reliable flow programming.
This solution requires designating a leaf as a border-leaf. The border-leaf is the leaf that is used to interconnect the
fabric to remote sites or a PIM router. The current software release supports a single border-leaf per site, however,
the border-leaf itself can have multiple layer 3 routed links towards the external network or sites.
The following CLI snippet shows the configuration of the Border Leaf:
! Specify the switch role under NBM controller configuration
nbm mode controller
switch-role border-leaf
!Enable PIM on the interface connecting remote site/external PIM router

description to_external_network
ip pim sparse-mode
ip router ospf 1 area 0
no shutdown
Figure 34. Multisite
Guidelines and Limitations

The following section presents guidelines and limitations for the solution described here.
Topology discovery
Because the topology can be discovered when the switches use POAP or when they are manually added to
DCNM, be sure to add only switches that are participating in the Cisco IP Fabric for Media and that are managed
by the DCNM media controller.
Flow setup
Flows can be set up using IGMP join messages and through an API call. The system supports IGMP Version 2
(IGMPv2) and IGMPv3. Also, a static OIF cannot be added through the CLI. Any static flow setup must be
implemented through an API call.
Note the following guidelines and limitations for flows set up using an API:
● DCNM notifies the broadcast controller and user about any unsuccessful API call that requires the
broadcast controller or user to retry the call. The notification is done via the AMQP bus.
● Static APIs are not supported when the receiver is connected through a Layer 2 port and Switch Virtual
Interface (SVI).
Host policy
The default host policy shipped with DCNM is a blacklist policy (allow all by default). Any changes to the default
policy must be made before any hosts are discovered on the system. The default policy cannot be edited after
hosts are active. This restriction does not apply to host-specific policies.
The receiver host policy applied to a host connected through a Layer 2 port and SVI applies to all the join
messages sent by all the hosts on that VLAN and cannot be applied to just a single receiver.
When there are multiple hosts behind a Layer 3 port or SVI and the default host policy is a whitelist, in certain
scenarios even when a host policy permits a host, ARP is the only mechanism through which the host is detected
and that specific host policy is programmed. Host detection through IGMP or multicast traffic may not work.
Do not modify any host policy from the CLI. You cannot apply any custom polices from CLI. Changes has to be
made using the DCNM media controller.
Flow policy
You must make any changes to the default policy before any flows are active on the fabric.
Flow policy cannot be modified while the flow is active. You can make modifications only when the flow ages out,
which typically occurs after the sender stops transmitting the flow (inactive flows time out after 180 seconds).
As a best practice, assume a flow size that is at least 5 percent greater than the flow bit rate. This configuration will
accommodate a certain amount of burst without causing the flow to be policed. For example, a 3-Gbps flow must
be provisioned as 3.15 Gbps or 3.3 Gbps.
Design
Always be sure that the uplink bandwidth from the leaf layer to the spine layer is equal to or greater than the
bandwidth to the endpoint. For example, when using a Cisco Nexus 93180YC-EX Switch, 48 x 10-Gbps
connectivity to the endpoints requires a minimum of 5 x 100-Gbps uplinks. You can use all 6 x 100-Gbps uplinks.
When possible, spread endpoint across different leaf switches to evenly distribute sources and receivers on all leaf
switches.
Limit the number of Spines to 2. In scenarios where a fixed SPINE like the 9326C is not sufficient, consider using
the Nexus 9508 as a SPINE.
Design redundant IP networks (2022-7). The networks should be independent, and each should have its own
instance of the DCNM media controller.
The bandwidth management algorithm does not account for unicast traffic on the network. Be sure that unicast
traffic, if any, occupies the minimum network bandwidth.
Live and File based workflows on same fabric

The benefit of a true IP Fabric is its ability to carry any IP Streams. Using QoS (Quality of Service) flows can be
prioritized according to business requirements. In an IP Fabric for Media, LIVE production workflows which are
multicast (2022-5, 2110) are of a higher priority than file based workflows, which are typically unicast. When
DCNM orchestrates multicast flows, it automatically puts the flows in high priority class, that way, as these flows
egress the switch, it is provided the highest priority. User has to define customer policies to ensure unicast flows
are placed in low priority queue. The configuration below ensures LIVE traffic (multicast) is prioritized over File
traffic (unicast).
! Create an ACL matching unicast traffic
ip access-list pmn-ucast
10 permit ip any 0.0.0.0 31.255.255.255
20 permit ip any 128.0.0.0 31.255.255.255
30 permit ip any 192.0.0.0 31.255.255.255
!Create an ACL matching multicast traffic
ip access-list pmn-mcast
10 permit ip any 224.0.0.0/4
policy-map type qos pmn-qos

class type qos pmn-ucast
set qos-group 0
class type qos pmn-mcast
set qos-group 7
class class-default
!Apply the service policy on all interfaces on all switches in the fabric
interface ethernet 1/1-54
service-policy type qos input pmn-qos
Single switch with controller

In certain deployments, a single switch provide sufficient ports and bandwidth and the endpoints are directly
connected to this switch. Any Nexus 9000 switch can be used with this type of deployment.
Controller offers features such as fabric configuration, flow visibility and analytics, security and north bound
integration with broadcast controller.
(Support for 9508 with R series line card will be available in a future release)
Setting up the NBM mode controller

! Enable the feature
feature nbm
feature nxapi
! Enable mode controller and specify controller IP

nbm mode controller
controller ip <dcnm_vip_address> vrf <vrf_name>
controller-credentials username admin password 0 <password>
! Verification
show nbm controller
VRF IP Role Status Online-

management 10.23.234.120 ACTIVE ONLINE 2016-12-04
14:43:00.151088
! Configuration below not required on R series Nexus 9500

Configuring interface
ip pim sparse-mode
ip pim passive
ip igmp version 3
no shutdown
Single switch without a controller

A single switch without controller can be deployed as well. There is no direct interaction between the broadcast
controller and the network. Broadcast controller communicate with the endpoints which using IGMP can subscribe
to any stream on the switch.
Below are the configurations required.

! Configuration not required on R series Nexus 9500

Configuring NBM mode flow

! 9500 with –EX series need the configuration below
! 9500 with R series line card is natively non-blocking and the CLI below is
optional
nbm mode flow
policy <policy-name>
bandwidth <flow-bandwidth>
ip group-range <ip-address> to <ip-address>
Configuring interface
ip pim sparse-mode
ip pim passive
ip igmp version 3
no shutdown
Conclusion
Cisco IP Fabric for Media provides the broadcast industry a path to migrate from SDI to an IP Based infrastructure
with multiple modes of deployment, be it a flexible Spine and leaf design or a single modular chassis or switch. The
fabric guarantees zero-drop multicast transport with minimal latency and jitter. The fabric is highly visible and
provides flow visibility and bit-rate statistics on a per-flow basis. Host polices help ensure that the fabric is secured
from any unauthorized endpoints. Integration with any broadcast controller through open REST APIs helps ensure
that any complexity in the network is abstracted and that the user has an unchanged operator experience.
For more information

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/ip_fabric_for_media/solution/guide_703i45/b_Cisco_Nexus_9000_Series_IP_Fabric_for_Media_Solution_Guide_
703I45.html.
https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-
735989.html.
736651.html.
738321.html.
Printed in USA C11-738605-02 04/18

Vxlan Design Guide

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Vxlan Design Guide

Caricato da

Copyright:

Formati disponibili

White Paper

Cisco IP Fabric for Media

Rahul Parameswaran (rparames)

An IP-based infrastructure with Cisco Nexus 9000 Series switches:

Figure 1(a) SDI router

Figure 1(b) IP fabric

Frame accurate switching

Nexus 9000 for IP fabric for Media

Table 1. Nexus 9000 switch port capacity and supported role

Part Number Description Supported Mode

N9K-C93180YC-EX and FX 48x10G/25G+ 6x100G Leaf with Controller/Standalone Switch

Spine and leaf design with network controller

Why use a layer 3 spine and leaf design

Non-blocking multicast in CLOS architecture

Why Media Fabrics can’t rely on ECMP and PIM alone

Flow orchestration with Cisco DCNM media controller

Cisco DCNM media controller features

Cisco DCNM media controller installation

● Number of 40-Gbps ports required: 40

Table 2. Supported leaf switch

Leaf Switch Endpoint Capacity Uplink Capacity

Cisco Nexus 9272Q Switch 36 x 40-Gbps endpoints 36 x 40-Gbps (1440-Gbps) uplinks

Cisco Nexus 93108TC-EX Switch 48 x 10-Gbps endpoints 6 x 100-Gbps (600-Gbps) uplinks

Spine Switch Number of Ports

Figure 3. Network topology with 9236c as spine

Fabric configuration using Power-On Auto Provisioning (POAP)

Figure 5. POAP process

When using POAP, you follow these steps:

● Create a DCHP scope for temporary IP assignment.

Figure 6. DCNM > Configure > POAP

Figure 7. DCNM > Configure > POAP > DHCP Scope

Figure 9. DCNM > Configure > POAP > POAP Definitions

Fabric configuration using the command-line interface

The configurations are shown here.

Setting up the NBM mode controller

! Enable mode controller and specify controller IP

VRF IP Role Status Online-

hardware access-list tcam region ing-racl 512

Configuring PIM and IGP (OSPF)

Configuring the interfaces

! Host Port Configuration (link between switch and end host)

interface Ethernet 1/2

interface vlan 100

vlan configuration 100

Figure 14(a) DCNM > Media Controller > Topology

Configuring the default host policy

Configuring the default flow policy

Flow visibility and bandwidth tracking

Flow statistics and analysis

Figure 20. DCNM > Media Controller > Flow Status

Figure 22. DCNM Flow View with Flow Alias

Events and notification

Precision Time Protocol

Quality of Service (QoS)

Service Class (Flow/Control) DSCP Values Queue

PTP Time and Sync EF Control/Priority Level 1

2022-6, 2110 Video AF41 Priority Level 2 or Bandwidth

API integration with the broadcast controller

For a list of APIs, see https://developer.cisco.com/site/data-center-network-manager/#overview.

The following types of failures may occur:

● Link flap between a leaf and spine (fabric link)

Cisco DCNM failure

Cisco DCNM media controller connectivity options: fabric control

Figures 27, 28, and 29 show the control options.