How to Multi-Home

Avi Freedman
VP Engineering
AboveNet Communications
 What is Multi-Homing?
• Multi-homing is the process of selecting,
provisioning, and installing a redundant
connection to the Internet.
• Could be the same provider, or a different
provider.
 Why Multi-Home?
• Slow is 1,000,000% better than dead.
• You may be out of bandwidth.
• And
– Telco circuits die.
– Routers die.
– Providers’ networks fail.
– Different networks have better performance to
different sites.
 A Multi-Homed Architecture
• Ideally, take advantage of the opportunity to
multi-home to remove all single points of
failure in your network.
• Use -
– Multiple providers, unless your current
provider will let you have cheap backup
– Multiple routers
– Multiple telco vendors
 Multi-Homed Architecture
• Two routers, each with a different WAN
connection from a different telco vendor.
• Use HSRP or VRRP internally to make
both routers look like one “virtual” router.
• Eventually, multiple providers.
• Upcoming Boardwatch article with configs.
 How the Internet Works
• Well, it breaks more than it works but when
it does work -
• The Internet is a network of networks.
• Each network (called Autonomous System)
on the Internet announces “routes”, which
are lists of the IP addresses of the boxes on
their network.
• You need to be able to send packets *to*,
and get packets *from*, everywhere.
 Inbound Traffic - Routes
• Routes are announced via BGP4 (the
Border Gateway Protocol)
• Routers are announced to BGP peers.
• Each “BGP peer” can be a “network peer”
or a “transit peer”.
• Network peers exchange just lists of
customer routes.
• Each route is tagged by the ASNs it passes
through.
 Inbound Traffic - Routes
• So when AboveNet and UUNET peer, only
AboveNet and UUNET routes are
exchanged. No Sprint, PSI, etc...
• Transit peers -
– Announce to their customers all of the routes
on the ‘net (AboveNet, UUNET, Sprint, PSI,
and the 60,000+ routes on the ‘net).
– Announce to their peers all routes heard via
transit.
 Inbound Traffic - Routes
• So if you advertise 207.106.96.0/19 to
AboveNet, -
– If you’re a network peer, they only re-announce
207.106.96.0/19 to customers (and use it
internally);
– If you’re a transit peer/customer, they announce
207.106.96.0/19 to all of their network peers.
• That’s how you get global *inbound*
reachability.
 Address Space Issues
• Noone wants to hear a route for you unless -
– You are multi-homed (even then, some people
don’t want to hear routers), or
– You have your own direct IP space allocation
from ARIN, RIPE, or APNIC.
• So, when you’re single-homed without your
own space, your IPs are reachable because
they’re part of your provider’s “aggregate”
block.
 Address Space Issues
• For example, your provider has
207.8.128.0/17.
• You have 27.8.197.0/24 from them.
• You’re single-homed.
• The only route on the ‘net for you is the
207.8.128.0/17 route, “originated” by your
provider’s ASN (and you don’t have to do
anything special).
 Address Space Issues
• If you have your own CIDR block and are
single-homed, your provider will originate
it.
• So, if you have 219.190.64.0/19, it’ll be
visible as an announcement by your
provider, originated into the BGP mesh
with your provider’s ASN as the “origin”.
 Address Space Issues
• If you have your own IP space and want to
multi-home, addressing issues are simple.
• Your other provider will start also
originating your IP blocks.
• Or you’ll start speaking BGP, originate
your IP blocks, and your providers will re-
advertise them to the world.
 Address Space Issues
• If you don’t have your own IP space, it’s a
bit more complicated.
• So, normally your ISP will only be
advertising 207.8.128.0/17 if you have
207.8.200.0/23.
• If you’re multi-homed, your other provider
will have to advertise 207.8.200.0/23.
• But *so will your first provider*.
• Why?
 Address Space Issues
• Routes are chosen first by specificity.
• That is, to how many IP addresses they
refer.
• The route “covering” the fewest IP is the
most specific, and wins.
• (Otherwise default would always win and
nothing would work.)
 Address Space Issues
• So, if ISP 1 advertises only 207.8.128.0/17
and ISP 2 advertises only 207.8.200.0/23,
all inbound traffic from the ‘net will come
in on ISP2.
• So, ISP 1 needs to “blow a hole in their
filters” to “leak” the more specific
207.8.200.0/23 route.
 Address Space: Filtering
• Some ISPs do or did filter on routes smaller
than (more specific than) /19s in >
205.0.0.0 space.
• But it doesn’t matter as long as your two
upstreams have good connectivity.
• Why?
 Address Space: Filtering
• If Sprint doesn’t see 207.8.200.0/23 from
ISP1 or ISP2, they’ll still see your
provider’s 207.8.128.0/17 route.
• So if your connectivity to ISP1 (the owner
of 207.8.128.0/17) goes down, all will be
well as long as ISP1 still sees
207.8.200.0/23 from ISP2.
• Sprint -> ISP1 -> ISP2
• This is why people don’t let you take IPs...
 Load-Balancing Outbound
• You can use static default routes to control
outbound packets.
– ip route 0.0.0.0 0.0.0.0 serial0/0
– ip route 0.0.0.0 0.0.0.0 serial1/0
• If they’re equal-cost (no metric at the end),
it’ll load-balance based on *destination*, by
default.
 Load-Balancing Outbound
• Why load-balance based on destination?
• For internal networking, sometimes per-
packet-load balancing makes sense.
• But if you’re trying to talk to England and
one provider has a 60ms path and the other
has a 150ms path, packets will arrive out of
order and TCP and UDP apps get unhappy
and slow.
 How it works, Single-Homed
• Outbound (easy):
– Use a default route to your provider.
• Inbound:
– Your provider originates a large (aggregate)
BGP route, and gives you some space from
inside it; and/or
– Your provider originates BGP routes for your
ARIN/RIPE/APNIC CIDR blocks as well.
 How it Works, Multi-Homed, Static
• Outbound (easy):
– Load-balance default routes to deal with
outbound packets.
• Inbound:
– Your providers both originate BGP routes for
just the address space you’re using, even if it’s
out of one provider’s space; and/or
– Your providers both originate BGP routes for
your ARIN/RIPE/APNIC CIDR blocks as well.
 How it Works, Multi-Homed, Static
• Special note:
– When providers configure BGP for single-
homed customers, they will generally “nail up”
your routes (even your directly-issued) CIDR
blocks, so that if your connection goes down
and up and down and ..., they don’t have to flap
that route out to the whole Internet. This is a
good thing.
 How it Works, Multi-Homed, Static
• Special note (ctd):
– But you NEED to make sure, when you’re
multi-homed, that the providers are NOT
nailing your routes up.
– Why?
– Because if they do, when one T1 goes down,
that provider will still advertise you to the
world, thus “blackholing” you.
 How it Works, Multi-Homed, BGP

• Topic of next talk.

• You either load-balance outbound with
statics, or take full routes from your
providers (if you can).
• You originate advertisements under your
ASN for your directly-issued CIDR blocks,
AND for the parts of your providers’ space
that you’re using (with their permission).
 The Transition: Static Routing
• To transition:
– Turn up the other T1/T3/Ethernet.
– Put IPs on the interface.
– Run tests end-end.
– Start load-balancing default to the new T1.
– Then, in the middle of the night, have the new
provider start advertising your IP space. Make
sure you have reachability to every other ISP
you can think of afterwards.
 The Transition: Static Routing
• To transition (ctd):
– After testing it live, turn off your other transit
pipes and make sure that, after a few minutes,
you still have connectivity.
 The Transition: BGP Routing
• To transition:
– Turn up the other T1/T3/Ethernet.
– Put IPs on the interface.
– Run tests end-end.
– Start load-balancing default to the new T1.
– Then, undo that and bring up a BGP session
that permits no routes either way.
– Then start taking routes, and watch outbound
traffic.
 The Transition: BGP Routing
• To transition (ctd):
– Then, start announcing your routes.
– Then, in the middle of the night, have your ISP
take out the static route and BGP
announcement they were making.
– Make sure your route is propagating.
– Test reachability.
– Turn off your other pipes.
– Test reachability.
 BGP or no?
• Advantages of doing static -
– Cheaper/smaller routers (less true nowadays)
– Simpler to configure
• Advantages of doing BGP -
– More control of your destiny (have providers
stop announcing you)
– Faster/more intelligent selection of where to
send outbound packets.
– Better debugging of net problems (you can see
the Internet topology now)
 Same Provider or Multiple?
• If your provider is reliable and fast, and
affordably, and offers good tech-support,
you may want to multi-home initially to
them via Frame, SMDS, or some backup
path (slow is 1,000,000% better than dead).
• Eventually you’ll want t multi-home to
different providers, to avoid failure modes
due to one provider’s architecture decisions.
 Questions?

• avi@freedman.net

• inet-access mailing list

• Nailing routes