Introduction To FRRouting

Introduction to FRRouting
Donald Sharp | Cumulus Networks

What are we going to cover?
● FRRouting
● Basic Routing with
○ BGP, OSPF and ISIS
○ VRF’s
● Advanced Data Center Routing
○ RFC 5549
○ VRF’s
○ BGP EVPN
● The Future!
What is FRRouting?
● Same routing capabilities of major vendors
○ Today development is supported by a large number of companies
■ https://frrouting.org
● Configuration similar to other vendors
○ Configuration file and interactive CLI available
● Runs natively on Linux and many other platforms
○ All features work on Linux, not true for other platforms
● Uses the Kernel’s routing Stack for packet forwarding
● GPLv2+
History of FRR
~1996 - Zebra development begins
~2002 - Quagga forked from Zebra
2016 - FRR forked from Quagga
Jan. 2017 - First Release - FRR 2.0
June 11, 2018 - Latest Stable - FRR 5.0

Why was FRRouting created?
● Wanted faster paced development
○ Needed ability to review in a cooperative way
○ Needed a CI system
○ Mistakes are ok in dev!
● Community consensus
● Under the auspices of the Linux Foundation
New FRRouting Features
● BABEL - RFC 6126
● EIGRP - RFC 7868
● NHRPD - RFC 2332 New Daemons!
● PBR
● LDPD - RFC 5561, 5918, 5919, 6667, 7473
● BGP - Large Communities, EVPN (type 2,3 and 5), RPKI, MPLS Based VPN’s, VPN/VRF Route Leaking, RFC
5549, Hostname Support, Shutdown Message, FlowSpec
● ISISD - RFC 5120(MT), SPF Backoff
● OSPF - Experimental SR, RFC 5549
● PIM - RFC 4611, 7761(Sparse Mode)
● VRF - supports both l3mdev and Namespace, not all daemons support VRF yet, BGP, ZEBRA, PIM, and OSPF
For a full list see the FRRouting wiki: https://github.com/FRRouting/frr/wiki
How To Setup/Where to Get
● Source - https://github.com/frrouting/frr
○ If Installing from source, please read http://docs.frrouting.org/projects/dev-guide/
● Releases - https://github.com/FRRouting/frr/releases
○ Can get both RPM and DEB packages for a wide variety of platforms here
Available Distributions/Builds On
● Debian
○ 8.1, 9.1
● Ubuntu
○ 12.04, 14.04, 16.04 and 18.04
● Redhat
○ 6, and 7
● BSD - FreeBSD 9,10, 11. NetBSD 6. OpenBSD 6
○ Caveats!
● Solaris(Omnios)
○ Caveats!
What kernel do you need?
● VRF - Depends on l3mdev
○ 4.4 for Limited Functionality, 4.8 for full functionality
● BGP EVPN - Depends on NTF_EXT_LEARNED and ARP Suppression
○ 4.17 and 4.14
● PIM-SM - Depends on IGMPMSG_WRVIFWHOLE and RTN_MULTICAST
netlink messages
○ 4.18
● These features are not currently available for non-Linux distributions
Starting/Stopping FRR
root@r1:/home/cumulus# systemctl start frr
● systemctl root@r1:/home/cumulus# vtysh
<start|restart|reload|stop> frr
Hello, this is FRRouting (version 4.0+cl3u2).
○ Does what is expected, reload Copyright 1996-2005 Kunihiro Ishiguro, et al.
applies new config written to
r1.rdu.cumulusnetworks.com# show ip route
/etc/frr/frr.conf Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
● vtysh T - Table, v - VNC, V - VNC-Direct, A - Babel, D -
○ This is the interactive shell SHARP,
F - PBR,
> - selected route, * - FIB route
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:00:06

C>* 10.50.11.0/24 is directly connected, eth0, 00:00:06
C>* 192.168.210.0/24 is directly connected, swp1, 00:00:06
C>* 192.168.240.1/32 is directly connected, lo, 00:00:06
Logging
● Where do my logs go? frr(config)# log file /tmp/frr.log
frr(config)# log syslog
○ show logging in vtysh
frr(config)# log stdout
○ 3 supported logging destinations: frr(config)#
1. Syslog - log syslog [level] frr# show logging
2. File - log file [name] Logging configuration for zebra:
3. Standard output - log stdout Syslog logging: level debugging, facility
● The log file has no output daemon, ident zebra
Stdout logging: level debugging
○ Designed to not be chatty unless Monitor logging: level debugging
debugs are turned on File logging: disabled
Protocol name: ZEBRA
○ If you are not sure what is going on use
Record priority: disabled
debug commands liberally Timestamp precision: 6
○ debug [protocol] ? if you are not sure
...
what to turn on.
FRR Architecture
BGP OSPF EIGRP RIP NHRP LDP
Zebra API
Zebra (RIB)
Netlink Bus Commands Netlink Bus Listen
Kernel (FIB)
ZAPI -vs- NETLINK
● Needed interprocess communication before NETLINK existed
● Have data communication needs that NETLINK doesn’t care about
○ ZEBRA_REDISTRIBUTE_XXX
○ ZEBRA_ROUTER_ID_XXX
○ ZEBRA_BFD_DEST_XXX
○ ZEBRA_NEXTHOP_XXX
○ ZEBRA_CAPABILITIES
● lib/zclient.[ch]
● Stream of data with each message type being sent as a packet
● Named Sockets
○ Only talk to FRR so we can change format as needed
Sources of Information
● Website
○ https://frrouting.org/
● Wiki
○ https://github.com/FRRouting/frr/wiki
● Documentation
○ http://docs.frrouting.org
● Email Aliases
○ https://lists.frrouting.org/listinfo
● GitHub
○ https://github.com/FRRouting
Basic Routing
Basic Routing
● Why?
● RIB -vs- FIB
● Metric
● Administrative Distance
● VRF
● Basic Topology and Config being used
● BGP Overview
● OSPF Overview
● IS-IS Overview
Why Routing?
● Networks are Complex beasts
○ static routes are not going to cut it in a complex network
● Not getting simpler with what people want to do with them
○ VM’s, Containers, etc.
● Routing allows us to control this chaos
○ Linux now has the ability to interact with the entirety of the network via a standards based
approach
BGP OSPF ISIS RIP NHRP LDP
RIB -vs- FIB
● Routing Information Base
○ A.k.a ‘Control Plane’ Zebra (RIB)
○ This is in FRRouting
● Forwarding Information Base
○ A.k.a ‘Data plane’ Kernel (FIB)
○ This is in the Kernel
FRR Kernel
robot# show ip route sharpd@robot ~> ip route show
Codes: K - kernel route, C - connected, S - static, R - RIP, default via 10.0.2.2 dev enp0s3 proto static metric 100
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, 4.3.2.1 dev enp0s8 scope link metric 14000
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 4.3.2.1 dev enp0s9 scope link metric 4278198272
F - PBR,
K * 0.0.0.0/0 [0/100] via 10.0.2.2, enp0s3 inactive, 01:24:15

D 4.3.2.1/32 [150/0] via 192.168.209.44, enp0s9, 00:00:02
K>* 4.3.2.1/32 [0/14000] is directly connected, enp0s8, 00:01:05
S 4.3.2.1/32 [1/0] is directly connected, enp0s10, 16:30:20
K * 4.3.2.1/32 [255/8192] is directly connected, enp0s9, 16:33:31
What is a Metric?
● Value used to figure out the ‘best’ route towards a destination
● Kernel has a metric it receives for a route
○ Assigned by the process that installs it
● Zebra receives metrics from each routing protocol for routes
Routing Protocol Measurement Range
RIP Hop Count 1-16
EIGRP Composite Metric 0-4.2 billion
OSPF Link State Distance > 100
● How do you determine the best metric to use?

Administrative Distance
/* From zebra/zebra_rib.c - Each route type's string
● Value assigned to Routing Protocols to * and default distance value.
*/
define precedence when comparing static const struct {
int key;
between them int distance;
} route_info[ZEBRA_ROUTE_MAX] = {
● Lower value wins [ZEBRA_ROUTE_SYSTEM] = {ZEBRA_ROUTE_SYSTEM, 0},
[ZEBRA_ROUTE_KERNEL] = {ZEBRA_ROUTE_KERNEL, 0},
● Default values assigned to each [ZEBRA_ROUTE_CONNECT] = {ZEBRA_ROUTE_CONNECT, 0},
[ZEBRA_ROUTE_STATIC] = {ZEBRA_ROUTE_STATIC, 1},
protocol [0-255] [ZEBRA_ROUTE_RIP] = {ZEBRA_ROUTE_RIP, 120},
[ZEBRA_ROUTE_OSPF] = {ZEBRA_ROUTE_OSPF, 110},
[ZEBRA_ROUTE_ISIS] = {ZEBRA_ROUTE_ISIS, 115},
[ZEBRA_ROUTE_BGP] = {ZEBRA_ROUTE_BGP, 20 /* IBGP is 200. */},
[ZEBRA_ROUTE_EIGRP] = {ZEBRA_ROUTE_EIGRP, 90},
[ZEBRA_ROUTE_NHRP] = {ZEBRA_ROUTE_NHRP, 10},
[ZEBRA_ROUTE_TABLE] = {ZEBRA_ROUTE_TABLE, 150},
[ZEBRA_ROUTE_LDP] = {ZEBRA_ROUTE_LDP, 150},
[ZEBRA_ROUTE_BABEL] = {ZEBRA_ROUTE_BABEL, 100},
/* no entry/default: 150 */
};
Admin Distance: Determining what to install
FRR Kernel
robot# conf t sharpd@robot ~> ip route show
robot(config)# ip route 4.3.2.1/32 enp0s10 default via 10.0.2.2 dev enp0s3 proto static metric 100
robot(config)# end 4.3.2.1 dev enp0s10 proto static metric 20
robot# show ip route 4.3.2.1 dev enp0s9 scope link metric 4278198272
Codes: K - kernel route, C - connected, S - static, R - RIP, 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, metric 100
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.208.0/24 dev enp0s8 proto kernel scope link src
F - PBR, 192.168.208.1 metric 100
> - selected route, * - FIB route 192.168.209.0/24 dev enp0s9 proto kernel scope link src
192.168.209.1 metric 100
K>* 0.0.0.0/0 [0/100] via 10.0.2.2, enp0s3, 00:04:13 192.168.210.0/24 dev enp0s10 proto kernel scope link src
S>* 4.3.2.1/32 [1/0] is directly connected, enp0s10, 00:00:04 192.168.210.1 metric 100
K * 4.3.2.1/32 [255/8192] is directly connected, enp0s9, 00:03:15
C>* 10.0.2.0/24 is directly connected, enp0s3, 00:04:13
robot# exit
How FRRouting interprets non-FRR route metrics
● Kernel has no concept of Admin Distance
● 4 Bytes of metric
○ High-Order byte becomes Admin Distance
○ Low-Order 3 bytes becomes Metric
● ip route add 4.3.2.1/32 dev enp0s9 metric 4278198272
robot# show ip route

Codes: K - kernel route, C - connected, S - static, R - RIP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 0xFF002000
F - PBR,
K>* 0.0.0.0/0 [0/100] via 10.0.2.2, enp0s3, 00:02:19

K>* 4.3.2.1/32 [255/8192] is directly connected, enp0s9, 00:01:21
Admin Distance and VRF default unreachable route
root@r1# ip rule show
● Want to terminate VRF route 1000: from all lookup [l3mdev-table]
32765: from all lookup local
lookup in the VRF table 32766: from all lookup main
32767: from all lookup default
root@r1# ip route show table 1001

● → VRF creation creates a default unreachable default metric 4278198272
unreachable route root@r1# vtysh -c "show ip route vrf DONNA"

Codes: K - kernel route, C - connected, S - static, R - RIP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
● Also want FRR to be able to install F
>
- PBR,
- selected route, * - FIB route
its own default route
○ Admin Distance! VRF DONNA:
K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable),
00:01:16
VRF Overview
● Separate tables for collections of different interfaces
○ L3 Segmentation
○ Separates routes learned from peers in each VRF
● Customers want areas of control
○ Separate telephone network from data network
○ PE providing routing for multiple customers
○ Security Segmentation
○ Overlapping IP’s from merging companies
https://www.netdevconf.org/1.1/proceedings/slides/ahern-vrf-tutorial.pdf
VRF Choices
● VRF Device
○ Creates a master device that you attach devices to
○ Route lookup by new rule inserted in kernel to allow usage of l3mdev to point at correct table
○ Allows route leaking via just installing a route with the appropriate outgoing nexthop, lookups
are easy from a RIB perspective
● Namespace
○ Create a network Namespace and insert devices into it
○ Route lookup by normal kernel table lookup
○ Route leaking can only be done by creating a new bridge and inserting taps into the
appropriate namespaces -or- running a physical cable from interfaces in different vrf’s
○ A heavyweight concept for table separation
○ Not actually a VRF, can be used like one
Basic Topology And Configuration
Topology Setup for Examples
192.168.211.0/24
r2 r3
192.168.210.0/24 192.168.212.0/24
192.168.213.0/24
r1 r4
192.168.215.0/24
192.168.214.0/24
r1 = .1 on all interfaces
r5 r3 = .3 on all interfaces
Loopbacks are 192.168.240.X/32
# r2 # r3
#The loopback network interface # The loopback network interface
auto lo auto lo
iface lo inet loopback iface lo inet loopback
address 192.168.240.2/32 address 192.168.240.3/32
auto swp1 auto swp1

iface swp1 iface swp1
address 192.168.210.2/24 address 192.168.211.3/24
auto swp2 auto swp2

iface swp2 iface swp2
address 192.168.211.2/24 address 192.168.212.3/24
# r1 # r4
# The loopback network interface # The loopback network interface
auto lo auto lo
iface lo inet loopback iface lo inet loopback
address 192.168.240.1/32 address 192.168.240.4/32
# r5 auto swp1
auto swp1
# The loopback network iface swp1
iface swp1
interface address 192.168.212.4/24
address 192.168.210.1/24
auto lo
iface lo inet loopback auto swp2
auto swp2
address 192.168.240.5/32 iface swp2
iface swp2
address 192.168.213.1/24 address 192.168.213.4/24
auto swp1
iface swp1 auto swp3
auto swp3
address 192.168.214.5/24 iface swp3
iface swp3
address 192.168.214.1/24 address 192.168.215.4/24
auto swp2
iface swp2
address 192.168.215.5/24
BGP Overview
● Used for routing on the Internet
○ Connects AS (Autonomous Systems) together
○ Policy Language is rich and featureful to allow operators a great level of control
● 2 modes of operation
○ IBGP (Route Reflector or Full Mesh)
○ EBGP
● Considered a path vector protocol
○ Uses AS Path to determine routes to install
○ AS Path also used for loop avoidance (don’t accept a path our our AS in it)
○ What route chosen is a complicated process and simplified here for purposes of discussion
● Uses TCP/IP for connections
● “BGP in the Data Center”, Dinesh Dutt (O’Reilly)
● https://tools.ietf.org/html/rfc1771 - This is the starter RFC, too many to list
BGP AS Path determination for Routing r1-r2
r2 r3
AS 10 AS 15
r1 r4
AS 5 AS 20
From r1:
To r2: (10),
r5
(20, 15, 10)
AS 25
(25, 20, 15, 10)
Shortest AS-Path Wins

r2 r3
AS 10 AS 15
r1 r4
AS 5 AS 20
From r1:
To r3: (10, 15)

r5
(20, 15)
AS 25
(25, 20, 15)
ECMP Shortest
AS-Path Wins
r2 r3
AS 10 AS 15
r1 r4
AS 5 AS 20
From r1:
r5 To r4: (20)
AS 25 (10, 15, 20)
(25, 20)

r2 r3
AS 10 AS 15
r1 r4
AS 5 AS 20
From r1:
r5 To r5: (25)
AS 25 (10, 15, 20, 25)
(20, 25)

BGP Basic Setup
! ● Neighbors
router bgp 5
bgp bestpath as-path multipath-relax
○ Who to peer with
neighbor 192.168.210.2 remote-as 10 ● Routes
neighbor 192.168.213.4 remote-as 20 ○ What routes do I originate?
neighbor 192.168.214.5 remote-as 25
!
■ Default is to originate nothing
address-family ipv4 unicast ○ Must either `redistribute XXX` or use a `network
redistribute connected A.B.C.D/M` statement
exit-address-family ■ Network statements only import existing prefixes
!
already existing in the RIB
■ To turn off this check use `no bgp network
import-check`
● Policy
○ Not discussing policy here but flexible
● multipath-relax allows ecmp to work with different AS-Paths but
the same length
● Each router needs to have the neighbor ip address set
appropriately
BGP Working Examples
r1.rdu.cumulusnetworks.com# show ip route root@r1:/home/cumulus# ip route show
Codes: K - kernel route, C - connected, S - static, R - RIP, default via 10.50.11.1 dev eth0
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, 10.50.11.0/24 dev eth0 proto kernel scope link src 10.50.11.194
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.210.0/24 dev swp1 proto kernel scope link src
F - PBR, 192.168.210.1
> - selected route, * - FIB route 192.168.211.0/24 via 192.168.210.2 dev swp1 proto bgp metric 20
192.168.212.0/24 via 192.168.213.4 dev swp2 proto bgp metric 20
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:41:06 192.168.213.0/24 dev swp2 proto kernel scope link src
C>* 10.50.11.0/24 is directly connected, eth0, 00:41:06 192.168.213.1
C>* 192.168.210.0/24 is directly connected, swp1, 00:41:06 192.168.214.0/24 dev swp3 proto kernel scope link src
B>* 192.168.211.0/24 [20/0] via 192.168.210.2, swp1, 00:13:44 192.168.214.1
B>* 192.168.212.0/24 [20/0] via 192.168.213.4, swp2, 00:13:25 192.168.215.0/24 proto bgp metric 20
C>* 192.168.213.0/24 is directly connected, swp2, 00:41:06 nexthop via 192.168.213.4 dev swp2 weight 1
B>* 192.168.215.0/24 [20/0] via 192.168.213.4, swp2, 00:10:53 192.168.240.2 via 192.168.210.2 dev swp1 proto bgp metric 20
* via 192.168.214.5, swp3, 00:10:53 192.168.240.3 proto bgp metric 20
C>* 192.168.240.1/32 is directly connected, lo, 00:41:06 nexthop via 192.168.210.2 dev swp1 weight 1
B>* 192.168.240.2/32 [20/0] via 192.168.210.2, swp1, 00:13:44 nexthop via 192.168.213.4 dev swp2 weight 1
B>* 192.168.240.3/32 [20/0] via 192.168.210.2, swp1, 00:10:53 192.168.240.4 via 192.168.213.4 dev swp2 proto bgp metric 20
* via 192.168.213.4, swp2, 00:10:53 192.168.240.5 via 192.168.214.5 dev swp3 proto bgp metric 20
B>* 192.168.240.4/32 [20/0] via 192.168.213.4, swp2, 00:13:25
B>* 192.168.240.5/32 [20/0] via 192.168.214.5, swp3, 00:13:16
BGP Debugging
● show bgp ipv4 unicast summary
○ Shows neighbor status and number of prefixes received from peers
● show bgp ipv4 unicast
○ Shows bgp routing table and best path selections
● debug bgp neighbor events
○ Indicates why peering is not coming up, look in log file
● debug bgp updates
○ Indicates what is happening for routes received, look in log file
My most common mistake is not setting up the peering relationship correctly

BGP Debugging Examples
r1.rdu.cumulusnetworks.com# show bgp ipv4 unicast summary
BGP router identifier 192.168.240.1, local AS number 5 vrf-id 0
BGP table version 11
RIB entries 11, using 1672 bytes of memory
Peers 3, using 58 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

r2.rdu.cumulusnetworks.com(swp1) 4 10 883 884 0 0 0 00:43:05 4
Total number of neighbors 3

BGP Debugging Examples Continued
frr# show bgp ipv4 uni
BGP table version is 11, local router ID is 192.168.240.1
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path

* 10.50.11.0/24 swp3 0 0 25 ?
* swp2 0 0 20 ?
* swp1 0 0 10 ?
*> 0.0.0.0 0 32768 ?
*> 192.168.240.1/32 0.0.0.0 0 32768 ?
* 192.168.240.2/32 swp2 0 20 15 10 ?
*> swp1 0 0 10 ?
* 192.168.240.3/32 swp3 0 25 20 15 ?
*= swp2 0 20 15 ?
*> swp1 0 10 15 ?
* 192.168.240.4/32 swp3 0 25 20 ?
*> swp2 0 0 20 ?
* swp1 0 10 15 20 ?
*> 192.168.240.5/32 swp3 0 0 25 ?
* swp2 0 20 25 ?
Displayed 6 routes and 15 total paths

OSPF Overview
● Considered an IGP (Interior Gateway Protocol)
○ Uses Link State Routing
■ Every router floods information, so that every router has all the information
○ Dijkstra’s Algorithm
● Uses IP (Layer 3) to advertise routing information
● OSPFv2 -vs- OSPFv3
○ IPv4 -vs- IPv6
● Supports multiple areas (and types of areas)
○ Allows Policy, Beyond Scope of talk
○ If you do more than one area you must have area 0
Dijkstra’s Example r1-r2
r2 60 r3
60 5
r1 100 r4
60 60 From r1:
To r2: (60)
r5 (100+5+60=165)
(60+60+5+60=125)
Numbers are example link costs

r2 60 r3
60 5
r1 100 r4
60 60 From r1:
r5 To r3: (100+5=105)
(60+60=120)
(60+60+5=125)

r2 60 r3
60 5
r1 100 r4
60 60 From r1:
r5 To r4: (100)
(60+60=120)
(60+60+5=125)

r2 60 r3
60 5
r1 100 r4
60 60 From r1:
To r5: (60)
r5
(100+60=160)
(60+60+5+60=185)

OSPF Basic Setup
# Put this in all FRR.conf r1.rdu.cumulusnetworks.com# show ip route root@r1:/home/cumulus# ip route show
! Codes: K - kernel route, C - connected, S - static, R - RIP, default via 10.50.11.1 dev eth0
router ospf O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, 10.50.11.0/24 dev eth0 proto kernel scope link src 10.50.11.194
network 0.0.0.0/0 area 0.0.0.0 T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.210.0/24 dev swp1 proto kernel scope link src
! F - PBR, 192.168.210.1
> - selected route, * - FIB route 192.168.211.0/24 via 192.168.210.2 dev swp1 proto ospf metric 20
192.168.212.0/24 via 192.168.213.4 dev swp2 proto ospf metric 20
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:07:24 192.168.213.0/24 dev swp2 proto kernel scope link src
O 10.50.11.0/24 [110/100] is directly connected, eth0, 00:07:24 192.168.213.1
C>* 10.50.11.0/24 is directly connected, eth0, 00:07:24 192.168.214.0/24 dev swp3 proto kernel scope link src
O 192.168.210.0/24 [110/100] is directly connected, swp1, 00:07:24 192.168.214.1
C>* 192.168.210.0/24 is directly connected, swp1, 00:07:24 192.168.215.0/24 proto ospf metric 20
O>* 192.168.211.0/24 [110/200] via 192.168.210.2, swp1, 00:05:34 nexthop via 192.168.213.4 dev swp2 weight 1
O>* 192.168.212.0/24 [110/200] via 192.168.213.4, swp2, 00:01:44 nexthop via 192.168.214.5 dev swp3 weight 1
O 192.168.213.0/24 [110/100] is directly connected, swp2, 00:07:24 192.168.240.2 via 192.168.210.2 dev swp1 proto ospf metric 20
C>* 192.168.213.0/24 is directly connected, swp2, 00:07:24 192.168.240.3 proto ospf metric 20
O 192.168.214.0/24 [110/100] is directly connected, swp3, 00:07:24 nexthop via 192.168.210.2 dev swp1 weight 1
O>* 192.168.215.0/24 [110/200] via 192.168.213.4, swp2, 00:00:12 192.168.240.4 via 192.168.213.4 dev swp2 proto ospf metric 20
* via 192.168.214.5, swp3, 00:00:12 192.168.240.5 via 192.168.214.5 dev swp3 proto ospf metric 20
O 192.168.240.1/32 [110/0] is directly connected, lo, 00:07:24
O>* 192.168.240.2/32 [110/100] via 192.168.210.2, swp1, 00:05:34
O>* 192.168.240.3/32 [110/200] via 192.168.210.2, swp1, 00:01:17
* via 192.168.213.4, swp2, 00:01:17
O>* 192.168.240.4/32 [110/100] via 192.168.213.4, swp2, 00:01:44
O>* 192.168.240.5/32 [110/100] via 192.168.214.5, swp3, 00:00:12
OSPF Debugging
● show ip ospf interface
○ What Interfaces is OSPF using?
● show ip ospf neighbor
○ What neighbors have I formed?
● show ip ospf database
○ Dump of the OSPF database
● show ip ospf database router
○ Dump of what each router sent you
● debug ospf packet
○ Look at data sent/received from peers, stored in log file
● debug ospf nsm
○ Look at neighbor state change events, stored in log file
OSPF Debugging Example
r1.rdu.cumulusnetworks.com# show ip ospf neigh
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
192.168.240.2 1 Full/Backup 39.325s 192.168.210.2 swp1:192.168.210.1 0 0 0
192.168.240.4 1 Full/Backup 35.278s 192.168.213.4 swp2:192.168.213.1 0 0 0
192.168.240.5 1 Full/Backup 31.861s 192.168.214.5 swp3:192.168.214.1 0 0 0
r1.rdu.cumulusnetworks.com# show ip ospf data
OSPF Router with ID (192.168.240.1)
Router Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum Link count

192.168.240.1 192.168.240.1 160 0x8000000c 0x8d15 5
192.168.240.2 192.168.240.2 286 0x80000009 0x2376 4
192.168.240.3 192.168.240.3 226 0x8000000a 0xa2e9 4
192.168.240.4 192.168.240.4 160 0x8000000c 0x7b09 5
192.168.240.5 192.168.240.5 123 0x80000009 0x2d4b 4
Net Link States (Area 0.0.0.0)
Link ID ADV Router Age Seq# CkSum

192.168.210.1 192.168.240.1 490 0x80000001 0xa366
192.168.211.2 192.168.240.2 286 0x80000001 0xa064
192.168.212.4 192.168.240.4 230 0x80000001 0x8d70
192.168.213.1 192.168.240.1 260 0x80000001 0x9e66
192.168.214.1 192.168.240.1 160 0x80000001 0xa161
192.168.215.4 192.168.240.4 160 0x80000001 0x8474
IS-IS Overview
● Considered a IGP (Interior Gateway Protocol)
● Uses Link State
○ Each Router floods their own link and interface information
○ Uses Dijkstra’s Algorithm to determine the shortest path tree
● Uses L2 to distribute information
○ Uses CLNP - ConnectionLess Network Protocol
● Easy to extend due to TLV support
○ Can carry `arbitrary` payloads
● Area’s are pre-defined
○ Level-1 - Individual Areas
○ Level-2 - Interconnecting different Areas
○ Level-1-2 - Border between 1 and 2
IS-IS Basic Setup
!
interface lo
● The `net XXX` statement is a bit of a holdover
ip router isis EVA from early IS-IS, each router needs a different
isis passive number
isis circuit-type level-1
! ● Need more config than OSPF because IS-IS
interface swp1 does both v4 and v6
ip router isis EVA
isis network point-to-point
!
interface swp2
ip router isis EVA
!
interface swp3
ip router isis EVA
!
router isis EVA
net 47.0023.0000.0000.0000.0000.0000.0000.1900.0004.00
metric-style wide
is-type level-1
!
IS-IS Configured
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.210.0/24 dev swp1 proto kernel scope link src 192.168.210.1
F - PBR, 192.168.211.0/24 via 192.168.210.2 dev swp1 proto isis metric 20
> - selected route, * - FIB route 192.168.212.0/24 via 192.168.213.4 dev swp2 proto isis metric 20
192.168.213.0/24 dev swp2 proto kernel scope link src 192.168.213.1
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:10:36 192.168.214.0/24 dev swp3 proto kernel scope link src 192.168.214.1
C>* 10.50.11.0/24 is directly connected, eth0, 00:10:36 192.168.215.0/24 proto isis metric 20
I 192.168.210.0/24 [115/20] via 192.168.210.2, swp1 inactive, 00:04:31 nexthop via 192.168.213.4 dev swp2 weight 1
I>* 192.168.211.0/24 [115/20] via 192.168.210.2, swp1, 00:04:31 192.168.240.2 via 192.168.210.2 dev swp1 proto isis metric 20
I>* 192.168.212.0/24 [115/20] via 192.168.213.4, swp2, 00:04:31 192.168.240.3 proto isis metric 20
I 192.168.213.0/24 [115/20] via 192.168.213.4, swp2 inactive, 00:04:31 nexthop via 192.168.210.2 dev swp1 weight 1
I 192.168.214.0/24 [115/20] via 192.168.214.5, swp3 inactive, 00:04:31 192.168.240.4 via 192.168.213.4 dev swp2 proto isis metric 20
C>* 192.168.214.0/24 is directly connected, swp3, 00:10:36 192.168.240.5 via 192.168.214.5 dev swp3 proto isis metric 20
I>* 192.168.215.0/24 [115/20] via 192.168.213.4, swp2, 00:04:31
* via 192.168.214.5, swp3, 00:04:31
I>* 192.168.240.2/32 [115/20] via 192.168.210.2, swp1, 00:04:31
I>* 192.168.240.3/32 [115/30] via 192.168.210.2, swp1, 00:03:55
* via 192.168.213.4, swp2, 00:03:55
I>* 192.168.240.4/32 [115/20] via 192.168.213.4, swp2, 00:04:31
I>* 192.168.240.5/32 [115/20] via 192.168.214.5, swp3, 00:04:31
IS-IS Debugging
● show isis interface
○ What interfaces is IS-IS using?
● show isis neighbor
○ What neighbors has IS-IS formed?
● show isis database
○ Dump the IS-IS database
● debug isis adj-packets
○ Debug IS-IS adjacency formation
● debug isis route-events
○ Debug IS-IS route related events
IS-IS Debugging Example
r1.rdu.cumulusnetworks.com# show isis neighbor
Area EVA:
System Id Interface L State Holdtime SNPA
r2.rdu.cumulusnetworks.comswp1 1 Up 28 2020.2020.2020
a.rdu.cumulusnetworks.com# show isis database
Area EVA:
IS-IS Level-1 link-state database:
LSP ID PduLen SeqNumber Chksum Holdtime ATT/P/OL
r4.rdu.cumulusn.00-00 155 0x00000004 0xee92 775 0/0/0
r1.rdu.cumulusn.00-00 * 155 0x00000006 0x7f11 745 0/0/0
r2.rdu.cumulusn.00-00 136 0x00000003 0x46e5 472 0/0/0
r5.rdu.cumulusn.00-00 136 0x00000003 0x6ab0 743 0/0/0
r3.rdu.cumulusn.00-00 136 0x00000004 0x2303 716 0/0/0
5 LSPs
What Routing Protocol to Use?
● Use what is most familiar and meets your needs
● BGP Scales better
○ This is mostly due to lack of link state flooding
● BGP can handle many afi/safi combinations that are missing in other routing
protocols
○ Can be used as both Overlay and Underlay for vpn networks
Which Routing Protocol should I use?
● PIM
○ If you want to do non Link Local Multicast Routing (224.0.0.0/4)
● RIP/EIGRP
○ RIP is ancient and limited in scope, EIGRP is not production ready
● NHRP
○ Wickedly complicated and has a special use case that is not common
● PBR
○ Policy Based Routing, limited use cases - “static routing with a twist”
● BABEL
○ Wireless mesh and home routing
● LDP
○ Label distribution for MPLS
What does FRR Provide?
● Provides ability to run routing anywhere in your network
○ Hosts
○ VM’s
○ Containers
● Connect to closed source vendors
○ Using standards based routing protocols
Advanced Data Center Routing
Advanced Data Center Design
● Modern Data Center Architecture
● RFC 5549
● BGP EVPN
Modern Data Center Architecture
SPINE ● What’s the Point of Clos?

○ Known Latency
○ Guaranteed Bandwidth
○ Easy to build
○ Limited Failure scope
● How to Connect Pods together
LEAF/TOR
○ Just add more Spines and call the
middle layers leafs!
● ECMP
○ No STP!
● MLAG is hard(L2 is evil)
RFC 5549 - The what and why
● Simplicity of Config for BGP
● V4 routes with v6 nexthops
● Ideal for datacenter point-to-point links
● Do not have to put a v4 address on every interface
● Commonly Referred to as ‘Unnumbered’
SPINE
● 45 ipv4 addresses down to 9

● Ratios change based upon the number
TOR
of links in your CLOS
FRR BGP/Linux Kernel Solution to 5549
● Have to cheat!
● Derive mac of peer from v6 RA interactions
● Install v4 nexthop using v4 LL space 169.254.0.1 and creating a neighbor
entry from derived mac.
Turn on RA for a interface

Send RA
Notification of LL Address Receive RA from Neighbor
Install Neighbor Entry

Form Peer over BGP Zebra Kernel
LL and Receive
Routes Install Route
Notice v6 LL Nexthop and
translate to 169.254.0.1
BGP 5549 Configuration
root@r1:/home/cumulus# cat /etc/network/interfaces
# The loopback network interface ● Interface based bgp peers
auto lo
iface lo inet loopback ● Causes BGP to turn on IPv6 Route Advertisement
address 192.168.240.1/32
auto swp2
for the interface.
iface swp2
● New keyword for neighbor statement
auto swp1
iface swp1 ○ remote-as <internal|external>
auto swp3
iface swp3 ● Allows cut ‘n paste BGP configuration for large
root@r1:/home/cumulus# vtysh -c ‘show run’ swaths of your DC
!
router bgp 5
bgp bestpath as-path multipath-relax
neighbor swp1 interface remote-as external
!
address-family ipv4 unicast
redistribute connected
!
BGP 5549 RIB -vs- FIB
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.240.2 via 169.254.0.1 dev swp1 proto bgp metric 20 onlink
F - PBR, 192.168.240.3 proto bgp metric 20
> - selected route, * - FIB route nexthop via 169.254.0.1 dev swp1 weight 1 onlink
nexthop via 169.254.0.1 dev swp2 weight 1 onlink
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:08:15 192.168.240.4 via 169.254.0.1 dev swp2 proto bgp metric 20 onlink
C>* 10.50.11.0/24 is directly connected, eth0, 00:08:15 192.168.240.5 via 169.254.0.1 dev swp3 proto bgp metric 20 onlink
B>* 192.168.240.2/32 [20/0] via fe80::a00:27ff:fe70:65bc, swp1, 00:06:19 root@r1:/home/cumulus# ip neigh show | grep 169.254
B>* 192.168.240.3/32 [20/0] via fe80::a00:27ff:fe70:65bc, swp1, 00:01:58 169.254.0.1 dev swp1 lladdr 08:00:27:70:65:bc PERMANENT
* via fe80::a00:27ff:fe42:f484, swp2, 00:01:58 169.254.0.1 dev swp2 lladdr 08:00:27:42:f4:84 PERMANENT
B>* 192.168.240.4/32 [20/0] via fe80::a00:27ff:fe42:f484, swp2, 00:01:58 169.254.0.1 dev swp3 lladdr 08:00:27:46:bc:5f PERMANENT
B>* 192.168.240.5/32 [20/0] via fe80::a00:27ff:fe46:bc5f, swp3, 00:00:08
OSPF Unnumbered
● Poorly Named
● Creates routes w/ ifindexes and onlink to get this to work.
r1# show run # The loopback network interface
interface swp1 auto lo
ip ospf area 0.0.0.0 iface lo inet loopback
● Note different config than earlier
ip ospf network point-to-point address 192.168.240.1/32 example
!
interface swp2 auto swp2 ● Must configure point-to-point
ip ospf area 0.0.0.0 iface swp2
ip ospf network point-to-point address 192.168.240.1/32
!
interface swp3 auto swp1
ip ospf network point-to-point address 192.168.240.1/32
!
interface lo auto swp3
! address 192.168.240.1/32
router ospf
!
https://docs.cumulusnetworks.com/display/DOCS/Open+Shortest+Path+First+-+OSPF+-+Protocol
OSPF Routing Results

T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, 192.168.215.0/24 via 192.168.240.4 dev swp2 proto ospf metric 20 onlink
F - PBR, 192.168.240.2 via 192.168.240.2 dev swp1 proto ospf metric 20 onlink
> - selected route, * - FIB route 192.168.240.3 proto ospf metric 20
nexthop via 192.168.240.2 dev swp1 weight 1 onlink
K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:07:33 nexthop via 192.168.240.4 dev swp2 weight 1 onlink
C>* 10.50.11.0/24 is directly connected, eth0, 00:07:33 192.168.240.4 via 192.168.240.4 dev swp2 proto ospf metric 20 onlink
O>* 192.168.215.0/24 [110/1767] via 192.168.240.4, swp2 onlink, 00:00:32 192.168.240.5 via 192.168.240.5 dev swp3 proto ospf metric 20 onlink
O 192.168.240.1/32 [110/0] is directly connected, lo, 00:02:36
C * 192.168.240.1/32 is directly connected, swp3, 00:07:33
O>* 192.168.240.2/32 [110/100] via 192.168.240.2, swp1 onlink, 00:02:59
O>* 192.168.240.3/32 [110/200] via 192.168.240.2, swp1 onlink, 00:00:34
* via 192.168.240.4, swp2 onlink, 00:00:34
O>* 192.168.240.4/32 [110/100] via 192.168.240.4, swp2 onlink, 00:00:34
O>* 192.168.240.5/32 [110/100] via 192.168.240.5, swp3 onlink, 00:00:07
r1.rdu.cumulusnetworks.com#
Why L2 is needed in a Data Center
● Legacy Apps still need L2 connectivity
○ Link Local Multicast
■ Service Discovery
○ Only have MAC address of whom to talk to
○ Desire to run applications on non-directly connected hosts
○ Assumption that IP addresses stay the same even when a endpoint is destroyed and
recreated
VxLan Encapsulation
● L2 Segmentation over a L3 network
○ Leverage all Data Center Links in the infrastructure
■ STP blocks links
○ Tunnels!
■ More complex control plane to allow higher availability
VxLan and BGP EVPN Setup
r2 r3
192.168.214.0/24
r1 r4
192.168.214.0/24
r5 r2 becomes the spine

r1 and r3 become leaf/tor with a
vxlan interface
r4 and r5 become hosts
# r1 # r2 # r3 # r4
# The loopback network interface #The loopback network interface # The loopback network interface # The loopback network interface
auto lo auto lo auto lo auto lo
iface lo inet loopback iface lo inet loopback iface lo inet loopback iface lo inet loopback
address 192.168.240.1/32 address 192.168.240.2/32 address 192.168.240.3/32 address 192.168.240.4/32
auto swp1 auto swp1 auto swp1 auto swp1

iface swp1 iface swp1 iface swp1 iface swp1
address 192.168.214.4/24
auto swp2 auto swp2 auto swp2
iface swp2 iface swp2 iface swp2
bridge-access 100
auto swp3
iface swp3 auto vxlan10100
bridge-access 100 iface vxlan10100 # r5
vxlan-id 10100 # The loopback network interface
auto vxlan10100 vxlan-local-tunnelip 192.168.240.3 auto lo
iface vxlan10100 vxlan-learning off iface lo inet loopback
vxlan-id 10100 bridge-access 100 address 192.168.240.5/32
vxlan-local-tunnelip 192.168.240.1
vxlan-learning off auto br1 auto swp1
bridge-access 100 iface br1 iface swp1
bridge-ports swp2 vxlan10100 address 192.168.214.5/24
auto br1 bridge-vids 100
iface br1 auto swp2
bridge-ports swp3 vxlan10100 iface swp2
bridge-vids 100 address 192.168.215.5/24
BGP EVPN - What is it?
● Simply Connect L2 networks across L3 Underlay Networks
● Unified Control Plane for VxLan
● Auto Distribution of MAC Addresses
○ Allows knowledge of where to send traffic
● Allows for MAC Mobility
https://cumulusnetworks.com/lp/evpn-data-center-oreilly/
https://www.netdevconf.org/2.2/slides/prabhu-linuxbridge-tutorial.pdf
#r2
BGP EVPN Setup !
router bgp 10
!
#r1
exit-address-family
!
!
router bgp 5
address-family l2vpn evpn
neighbor swp1 activate
!
exit-address-family
!
exit-address-family
!
neighbor swp1 activate #r3
advertise-all-vni !
exit-address-family router bgp 15
! neighbor swp1 interface remote-as 10
!
exit-address-family
!
advertise-all-vni
exit-address-family
!
BGP EVPN Configured
r1.rdu.cumulusnetworks.com# show bgp l2vpn evpn summ
BGP table version 0

BGP table version 0

a.rdu.cumulusnetworks.com(swp1) 4 5 1794 1796 0 0 0 01:25:00 2

BGP table version 0


BGP EVPN Configured
r1.rdu.cumulusnetworks.com# show bgp l2vpn evpn
Route Distinguisher: ip 192.168.240.1:2
● Type 2 route is the MAC route
*> [2]:[0]:[0]:[48]:[08:00:27:46:bc:5f] ● Type 3 route specifies how to
192.168.240.1 32768 i
*> [3]:[0]:[32]:[192.168.240.1] send BUM Traffic
192.168.240.1 32768 i
Route Distinguisher: ip 192.168.240.3:2
*> [2]:[0]:[0]:[48]:[08:00:27:70:38:6e]
192.168.240.3 0 10 15 i
*> [3]:[0]:[32]:[192.168.240.3]
192.168.240.3 0 10 15 i
Displayed 4 out of 4 total prefixes

r1.rdu.cumulusnetworks.com# show evpn mac vni all
VNI 10100 #MACs (local and remote) 2
MAC Type Intf/Remote VTEP VLAN

08:00:27:46:bc:5f local swp3
08:00:27:70:38:6e remote 192.168.240.3
Future Enhancements
Future Enhancements
● Zebra rewrite to further use nexthop-groups
○ To match http://vger.kernel.org/netconf2017_files/nexthop-objects.pdf
● CLI Rewrite(Again!) to take advantage of netconf/yang
● Full Remote DataPlanes
● More RFC’s than you can shake a stick at
○ https://github.com/FRRouting/frr/wiki/Feature-Requests
○ Type-4 EVPN
Thanks!
Super Special Thanks to Roopa Prabhu, David Ahern and Nikolay Aleksandrov for putting up with all my kernel questions!
Thanks to Don Slice, Quentin Young, and Eric Pulvino for talking me down when I worked on this Presentation

Introduction To FRRouting

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Introduction To FRRouting

Caricato da

Copyright:

Formati disponibili

Introduction to FRRouting

Donald Sharp | Cumulus Networks

~2002 - Quagga forked from Zebra

2016 - FRR forked from Quagga

Jan. 2017 - First Release - FRR 2.0

June 11, 2018 - Latest Stable - FRR 5.0

K>* 0.0.0.0/0 [0/0] via 10.50.11.1, eth0, 00:00:06

BGP OSPF EIGRP RIP NHRP LDP

Netlink Bus Commands Netlink Bus Listen

K * 0.0.0.0/0 [0/100] via 10.0.2.2, enp0s3 inactive, 01:24:15

Routing Protocol Measurement Range

RIP Hop Count 1-16

EIGRP Composite Metric 0-4.2 billion

OSPF Link State Distance > 100

● How do you determine the best metric to use?

robot# show ip route

K>* 0.0.0.0/0 [0/100] via 10.0.2.2, enp0s3, 00:02:19

root@r1# ip route show table 1001

unreachable route root@r1# vtysh -c "show ip route vrf DONNA"

auto swp1 auto swp1

auto swp2 auto swp2

Shortest AS-Path Wins

To r3: (10, 15)

Shortest AS-Path Wins

Shortest AS-Path Wins

My most common mistake is not setting up the peering relationship correctly

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

Total number of neighbors 3

Network Next Hop Metric LocPrf Weight Path

Displayed 6 routes and 15 total paths

Numbers are example link costs

Numbers are example link costs

Numbers are example link costs

Numbers are example link costs

r1.rdu.cumulusnetworks.com# show ip ospf data

OSPF Router with ID (192.168.240.1)

Router Link States (Area 0.0.0.0)

Link ID ADV Router Age Seq# CkSum Link count

Net Link States (Area 0.0.0.0)

Link ID ADV Router Age Seq# CkSum

SPINE ● What’s the Point of Clos?

● 45 ipv4 addresses down to 9

Turn on RA for a interface

Notification of LL Address Receive RA from Neighbor

Install Neighbor Entry

r1.rdu.cumulusnetworks.com# show ip route root@r1:/home/cumulus# ip route show

r5 r2 becomes the spine

auto swp1 auto swp1 auto swp1 auto swp1

Total number of neighbors 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

Total number of neighbors 2

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

Total number of neighbors 1

Displayed 4 out of 4 total prefixes

VNI 10100 #MACs (local and remote) 2

MAC Type Intf/Remote VTEP VLAN

Potrebbero piacerti anche