VXLAN

By Anand Nande
AGENDA
○ What is VXLAN ?
○ Why VXLAN ?
○ How does it work ?
○ So now we can migrate VMs across subnets?
○ What about routing across VXLANs?
○ Any Performance Impact?
○ Demo
What is it ?
Virtual eXtensible Local Area Network
● Tunneling protocol co-developed by a group of companies.

● Original description as per the RFC:

“ Used to address the need for overlay networks

within virtualized data centers accommodating multiple tenants. The
scheme and the related protocols can be used in networks for cloud
service providers and enterprise data centers ”

● Encapsulation method to extend L2 traffic over a L3 network. VXLAN uses

multicast in the backend. It uses traditional flood and learn for MAC learning
and Address Resolution Protocol (ARP) resolution.
Why VXLAN?
What problems does it address and how am I selling it
It's intended to solve the following issues
● Limitations Imposed by Spanning Tree and VLAN Ranges
○ STP blocks the use of links to avoid the replication and looping of frames.
■ This is a prob to some DC admins who pay for ports and links
■ Resiliency due to multipathing - not available.
○ 2^12 = 4096
■ A 12-bit VLAN ID used to divide multiple broadcast domains
■ STP blocks few id’s from this domain
● Multi-tenant Environments
○ Is it possible to address need for multiple VLANs per tenant w/ 4096 limit?
○ L3 networks not a comprehensive solution. 2 tenants might use the same set of Layer 3
addresses within their networks.
● Inadequate Table Sizes at ToR Switch
How does it work?
Let’s not draw conclusions yet..there’s a lot more to it
● VXLAN encapsulates Layer 2 frames into Layer 3
packets (using UDP)

● Adds a 24-bit VXLAN Network Identifier (VNI) that allows

for up to 16 million unique combinations

● VXLAN Segments are built between VXLAN Tunnel

Endpoints (VTEPs)...more on this in coming slides

● Not understood by any physical networking devices (the

transport that carries the encapsulated frames only needs
an IP-based network).
VXLAN Encapsulation

The L2 Ethernet Frame

The Final packet that goes out on L-3.

Note how inner-MAC is payload here
and outer-MAC + UDP is added.

The VNI is the only distinguished

identifier for a VXLAN segment
 VTEPs across a L3 network

Hypervisor 1 Hypervisor 2

10.0.0.1 10.0.0.2

VM2-1
VM1-1 172.16.1.0/24 VNI=10
VTEP-1 VTEP-2
IP=172.16.1.12
192.168.1.0/24 VNI=20
IP=172.16.1.10 MAC=52:54:00:30:de:e3
MAC=52:54:00:0e:08:b3 VNI=10
VNI=10

L3 Network

VM1-2 VXLAN segment-1 w/ VNI=10 VM2-2

IP=192.168.1.100 VXLAN segment-2 w/ VNI=20 IP=192.168.1.111

MAC=00:0C:29:2F:32:A0 MAC=00:0C:29:2F:23:A0
VNI=20 VNI=20
VTEPs responsible for
encap/decap packets at
both ends of the wire
 VXLAN Multicast group
VTEP-3s table
MAC VNI ID REMOTE VTEP

52:54:00:a0:1b:bb 10 192.168.122.186

52:54:00:8a:bd:ff 10 192.168.122.101
VTEP-1s table
MAC VNI ID REMOTE VTEP

52:54:00:60:18:f9 10 192.168.122.141

52:54:00:8a:bd:ff 10 192.168.122.101

MAC VNI ID REMOTE VTEP

52:54:00:a0:1b:bb 10 192.168.122.186

VTEP-2s table
52:54:00:60:18:f9 10 192.168.122.141
packet
capture
on one
of the
VTEP’s
Additional
Wireshark
plugin
required to
analyse the
UDP data here
So now can we migrate
VMs across subnets?
..the answer is ‘YES - we can’
Whats the most important thing when we migrate a VM ?
- VM’s ip and mac addresses should not change even if the VM has been
migrated to a new hypervisor.
- We use the reference controller which learns the new VM-placement using
ovsDB.
# ps -ef | grep controller | grep -v color
root 1396 1325 0 Feb17 pts/3 00:00:01 controller -v
ptcp:6633

- This reference controller is located on each VTEP.

 VM communication via VTEP post-migration

Hypervisor 1 Hypervisor 2

10.0.0.1 10.0.0.2

VM2-1
Flows related to 172.16.1.0/24 VNI=10
vm1-1 removed on VTEP-1 VTEP-2
VTEP-1 and added to IP=172.16.1.12
VTEP-2 MAC=52:54:00:30:de:e3
VNI=10

* The VM's port is updated to

point to the new host.

* When the port is updated by

VM1-2 Nova, Neutron executes ML2 VM1-1
mechanism drivers, including
IP=192.168.1.100 l2pop, which then sends RPC IP=172.16.1.10
MAC=00:0C:29:2F:32:A0 messages to the new host. MAC=52:54:00:0e:08:b3
VNI=20 VNI=10
What about routing across
VXLANs ?
 Lets look at the 2 way communication that happens
physical:virtual virtual:physical

[VXLAN-to-VLAN]
[VLAN-to-VXLAN]
- vlan to router
- also known as ‘vxlan-bridging’ that extends
the l2-domain over a vast l3-network. - hardware VTEP(router) only does bridging

- VTEP’s are responsible in this case to the - on physical router:

check for the VNI id in the VXLAN header. If
it matches the one of the VNI’s thats in its - in_ports, out_ports, loopback_ports
table, forward the packet to the relevant
1. in_ports > bridge into a vlan > lo_ports
vxlan-segment to take care of it.
2. router_fib > next_hop > physical_world
- Switch should be capable of vxlan handling.
- performance loss because of loopback
Any Performance Impact?
cliche but true “Your loss is someone else's gain”
● An overlay network is simply a computer network which is built on top of another network.

● The added flexibility comes at a cost, due to the additional processing overhead for encapsulation
and de-encapsulation of packets. This consumes both CPU resources and degrades network
performance, especially for high speed connections.

● By introducing hardware offloading capabilities that can be found in some of today’s modern NICs,
the added overhead for packet processing can be offloaded to the NIC hardware, resulting in
improved CPU utilization and higher throughput.

● Improve performance of VXLAN w/ hardware offload: https://access.redhat.com/articles/1390483

 Demo
Let’s put this into action by creating a simple topology
 192.168.122.141
192.168.122.186 192.168.122.101

Ignore for the purpose of demo.

You can do this later as an advanced exercise
Hands on
[On your KVM host or laptop]
# yum install virt-manager
libvirt* qemu-kvm openvswitch
# cd /var/lib/libvirt/images
## Get the mininet vmdk image
from the mininet website
## write the spawn.sh
# chmod +x spawn_vm.sh
# ./spawn_vm.sh <vmname>
(note you need to spawn 2
mininet VMs, hence use the above
script 2wice)
# cat > spawn.sh << EOM
> #!/bin/sh
> export VM_NAME=$1
> export portgroup=$2
> IMAGES_BASE=/var/lib/libvirt/images
> cp $IMAGES_BASE/mininet.vmdk
$IMAGES_BASE/$VM_NAME.vmdk
>
> virt-install -r 256 \
> -n $VM_NAME \
> --vcpus=1 \
> --import \
> --autostart \
> --memballoon virtio \
> --network network=host-only\
> --disk $IMAGES_BASE/$VM_NAME.vmdk \
> EOM
[Inside your Mininet VM-1]

Create a simple topology:

mininet@mininet-vm:~$ sudo mn --topo single,1
Create vxlan interface on each side:
mininet> sh ovs-vsctl add-port s1 vxlan
Setup vxlan interface on each side
mininet> sh ovs-vsctl set interface vxlan type=vxlan option:
remote_ip=192.168.122.101
Setup IP for hosts:
mininet> h1 ifconfig h1-eth0 10.0.0.1

[Inside your Mininet VM-2]

mininet@mininet-vm:~$ sudo mn --topo single,1

mininet> sh ovs-vsctl add-port s1 vxlan

mininet> sh ovs-vsctl set interface vxlan type=vxlan option:

remote_ip=192.168.122.186

mininet> h1 ifconfig h1-eth0 10.0.0.2

Advanced VXLAN configuration
Manually assign VXLAN Network ID (VNI) and/or OpenFlow port number

e.g. VNI=20, OF_PORT=9

# ovs-vsctl set interface vxlan type=vxlan option:remote_ip=192.168.122.186

option:key=5566 ofport_request=9

Assign VNI flow by flow (Useful when implementing multi-tenant environment)

# ovs-vsctl set interface vxlan type=vxlan option:remote_ip=140.113.215.200

option:key=flow ofport_request=9

and then setup your flow entries by

# ovs-ofctl add-flow s1 in_port=1,actions=set_field:5566->tun_id,output:9

# ovs-ofctl add-flow s1 in_port=9,tun_id=5566,actions=output:1

References
- IETF RFC : tools.ietf.org/html/rfc7348
- Scott Lowe’s blog : http://blog.scottlowe.org/
- David Mahlers youtube channel : https://www.youtube.com/user/mahler711
- VMware VXLAN Performance Evaluation : https://www.vmware.com/files/pdf/techpaper/VMware-
vSphere-VXLAN-Perf.pdf
- TCP/IP over VXLAN Bandwidth Overheads: http://packetpushers.net/vxlan-udp-ip-ethernet-
bandwidth-overheads/