Carnegie Mellon University

Research Showcase
CyLab Research Centers and Institutes

12-28-2010

SCI-FI: Domain-based Scalability, Control and

Isolation for the Future Internet (CMU-
CyLab-10-020)
Hsu-Chun Hsiao
Carnegie Mellon University, hsuchunh@andrew.cmu.edu

Xin Zhang
Carnegie Mellon University, xzhang1@cmu.edu

Geoff Hasker
Carnegie Mellon University, hasker@cmu.edu

Haowen Chan
Carnegie Mellon University, haowenchan@cmu.edu

Adrian Perrig
Carnegie Mellon University, perrig@cmu.edu

See next page for additional authors

Recommended Citation
Hsiao, Hsu-Chun; Zhang, Xin; Hasker, Geoff; Chan, Haowen; Perrig, Adrian; and Andersen, David G., "SCI-FI: Domain-based
Scalability, Control and Isolation for the Future Internet (CMU-CyLab-10-020)" (2010). CyLab. Paper 78.
http://repository.cmu.edu/cylab/78

This Technical Report is brought to you for free and open access by the Research Centers and Institutes at Research Showcase. It has been accepted
for inclusion in CyLab by an authorized administrator of Research Showcase. For more information, please contact kbehrman@andrew.cmu.edu.
Authors
Hsu-Chun Hsiao, Xin Zhang, Geoff Hasker, Haowen Chan, Adrian Perrig, and David G. Andersen

This technical report is available at Research Showcase: http://repository.cmu.edu/cylab/78

SCI-FI: Domain-based Scalability, Control and Isolation
for the Future Internet

Hsu-Chun Hsiao, Xin Zhang, Geoff Hasker, Haowen Chan, Adrian Perrig, David Andersen

December 28, 2010

CMU-CyLab-10-020

CyLab
Carnegie Mellon University
Pittsburgh, PA 15213
 SCI-FI: Domain-based Scalability, Control and Isolation for the Future
Internet
Hsu-Chun Hsiao Xin Zhang Geoff Hasker Haowen Chan Adrian Perrig
David Andersen
CyLab / Carnegie Mellon University
28 December 2010

Abstract We show that a high level of control and isolation nat-

urally leads to security and reliability without the use of
We present the first Internet architecture designed for high-overhead security mechanisms, while exposing ex-
control and isolation. We propose to separate ASes into pressive and diverse communication path sets to the end-
groups of independent routing sub-planes which then in- points.
terconnect to form complete routes. Our architecture,
SCI-FI, provides superior resilience and security proper- In particular, we introduce the abstract notion of a hi-
ties as an intrinsic consequence of good design principles, erarchy of trust domains whose members all share a com-
without needing additional add-on protocols or external mon contractual, legal, cultural, geographical, or other
checks to provide resilience. Our security analysis shows basis for extending limited trust between each other. Ex-
that SCI-FI can naturally prevent several long-standing amples may be a domain of U.S. educational institutions,
security plagues to existing interdomain routing proto- ISPs that participate in the same peering point who share
cols even with their semantics perfectly secured. Our a common, binding legal contract on their behavior, or
evaluation results further demonstrate SCI-FI’s routing ISPs in the same state or country who are subject to
efficiency, path expressiveness, and substantial reliability the same laws and regulations. Using this abstraction,
improvements over existing (secured) routing protocols in we provide the machinery to guarantee a critical isolation
the presence of malicious attacks. property at the network level: Entities outside a trust do-
main cannot affect communication within that trust do-
main. For communication that must span trust domains,
we provide the property that limits the entities who can
1 Introduction affect the communication to a necessary and explicitly
identified set of other trust domains.
The Internet is the most geographically, administratively,
and socially diverse distributed system ever invented. Finally, SCI-FI assigns control of route selection to the
While today’s Internet architecture admits some admin- source and destination. The source and destination agree
istrative diversity, such as by separating routing inside a jointly on which path to use from the set of ISP-supported
domain (intra-AS routing) from the global inter-domain routes. The architecture naturally controls routing infor-
routing, it falls short in handling key challenges of se- mation flow, and provides for explicit trust in path selec-
curity and isolation that arise in this intensely heteroge- tion.
neous setting. As a result, we see surprisingly frequent
incidents in which communication is interrupted by ac- Contributions. We design and analyze an Internet ar-
tions or actors far from the communicating entities: In chitecture emphasizing the principles of control and iso-
addition to classical examples such as YouTube being lation. The resulting architecture enables route control
globally disrupted by routing announcements from Pak- for ISPs, senders and receivers at an appropriate level
istan [1], other issues surrounding the lack of resource of granularity, balancing efficiency, expressiveness, policy
control and isolation are not solved by existing propos- compliance, and security. The isolation properties dra-
als such as S-BGP: The introduction of excessive routing matically reduce the TCB and provide situational aware-
churn [2]; traffic flooding; router memory resource ex- ness which entities need to be relied upon for various net-
haustion; and even issues of global conflicts over naming work operations. We find that the resulting architecture
and name resolution. offers strong security properties and demonstrate that the
In this paper, we propose a clean-slate Internet archi- resulting routes are comparable to the routes that BGP
tecture that provides strong guarantees for isolation and uses. We anticipate that the proposed architecture of-
control in ways that map well to existing geographic, po- fers a useful point in the design space for the search of
litical, and legal boundaries. next-generation Internet routing protocols.

1
2 Limitations of Current Routing
Design
To motivate our new design, we first demonstrate the fun-
damental limitations in the current inter-domain routing
protocols. Through concrete examples and discussion,
we show that recent popular inter-domain routing pro-
tocols [3, 4, 5, 6, 7, 8, 9, 10], even with their semantics
perfectly secured (e.g., via S-BGP [11] like approaches),
do not provide several important security properties.
Limitation 1: Arbitrary Information Flow. We
first show that existing inter-domain routing protocols
give endpoints and ISPs little control over how their rout-
ing announcements are being further propagated, which
can cause several security vulnerabilities. More specifi-
cally, path-vector routing is used by many inter-domain
routing designs due to its ability to support rich rout-
ing policies [12]. For example, in addition to the current
de facto inter-domain routing protocol BGP, MIRO [5],
R-BGP [6], Route Deflection [8], and ACR [9] use path-
vector routing as the underlying vehicle for route dissem-
ination; and Pathlet routing still uses BGP for dissemi-
nating the “pathlets”. In these routing systems, however,
once a certain node N announces its prefix, or path, or
pathlet to its neighbors, N has no control over the way in
which its routing update is further propagated and paths
are constructed as they flow towards or away from N .
Figure 1(a) depicts an example scenario. A destination
AS 1 is served by a provider AS 2; likewise the source
AS (AS 5) is served by its provider AS 4. An interme-
diate AS 3 peers with both providers; the providers do
not peer with each other directly. Suppose that AS 3 is
malicious or compromised and wishes to control the route
between the source and destination. It can forward the
route it learns from AS 2 to AS 4 even if S-BGP security
is used. Because routes announced by peers are gener-
ally preferred over routes announced by providers, this
will likely result in the destination using the AS-PATH
{1, 2, 3, 4, 5}, which violates the valley-free routing prin-
ciple [13]. Using conventional routing security measures,
such as S-BGP, (which only secures the strict semantics
of path vector) it is impossible to distinguish this route
from a legitimate route. Currently, the only practical
method for dealing with such anomalies is to use hand-
tuned ingress or egress filters to custom-configure the sys-
tem, which can be extremely error-prone and cause incon-
sistencies [14] [15].
Figure 1(b) depicts another example where the end-
point AS E is the destination of traffic. AS E generates a
route advertisement for its address prefix which is propa-
gated through its provider A; this advertisement is further
propagated into separate paths P1 and P2 going through
B and M , respectively, and re-converging at AS C. Sup-
pose AS C selects P2 to re-advertise; then P1 is discarded
and all inbound traffic to E now must pass through the
AS M which is less preferred by E.
2
...
...
Compromised
AS
C
Peering Peering
4 3 2 B M
Source’s Dest’s P1
A P2
Provider AS Provider AS
5 1
E
Source AS Dest AS
(a) Valley-free violation. (b) Lack of inbound
traffic control.
Figure 1: Arbitrary information flow. Small arrows indi-
cate customer-provider (downstream/upstream) relation-
ships. Large arrows indicate constructed paths. In each
case, P1 is preferred by the endpoint but P2 is the path
that is used.
In summary, routing systems with one-directional and
unregulated flow of routing update dissemination can suf-
fer from the following problems: (i) paths can be con-
structed from provider to customer and back to a different
provider (a “valley” situation); (ii) paths can traverse un-
trusted ISPs; and (iii) the routing system is generally sub-
ject to arbitrary blackhole and wormhole attacks. This
unprincipled manner of path construction is a well-known
source of persistent Internet route fluctuation [13].
Limitation 2: No Joint Path Selection between
Source and Destination. We argue that the lack of
joint path selection between the source and destination
nodes prevents effective defenses against long-standing
Denial-of-Service attacks. More specifically, traditional
path-vector routing places more control to the intermedi-
ate ASes: after receiving advertisements from their neigh-
bors, each AS selects its most-preferred routes which are
then advertised to its peer and customer ASes. In such
a system, endpoints have no control over path construc-
tion. Newer proposals for multi-path routing [3, 4, 5, 8]
recognize that the users of a path – the communicating
endpoints – should have the final say over whether a route
is acceptable. These proposals give the source the abil-
ity to select from a set of diverse paths, but they do not
similarly empower the destination to control its inbound
traffic, as illustrated in Figure 1(b). Consequently, the
routing infrastructure provides no built-in mechanisms for
a destination to block unwanted traffic early in the net-
work before it reaches the destination. Without inbound
traffic control, a destination lacks the intrinsic ability to
defend against malicious incoming traffic such as Denial-
of-Service attacks.
Limitation 3: Lack of Routing Isolation. A cen-
tral tenet of current inter-domain routing architectures
has been to achieve global reachability, where a routing
announcement or prefix advertisement from a certain AS
can potentially be propagated throughout the entire In-
ternet. In other words, most (if not all) ASes are in the
 Inter-top
same flat routing dissemination domain. For example, in TD route
addition to the aforementioned multipath routing proto- Top-level
Top-level
D
cols, NIRA [16] organizes all the ASes in one tree-based Trust Domain In est.
Trust Domain
d gr Co
routing domain, and Landmark routing [17] also makes te es n
lec s r tro
- se te ou ll
the routing “landmarks” available throughout the net- ce u te ed
ur s ro
So res
work. While global visibility simplifies achieving global Eg Local
Local Trust Domain
routing reachability, it also lends great help to individual Trust Domain
malicious ASes which can easily launch attacks influential Source
Destination
to the entire Internet. For example, two distant collud- “Localized” within each TD
Strict Inter-TD Isolation
ing ASes can announce a (non-existing) wormhole link
between each other to create a (bogus) short path, which
can be seen potentially by the entire Internet and thus Figure 2: Proposed Internet Architecture
attract a considerable amount of traffic.
Limitation 4: Lack of Route Freshness. In today’s well-specified influence perimeter, knowing exactly which
routing protocols, an adversary who can cause messages other domains hold influence over their routing and for-
to be delayed or dropped can force traffic to continue to
warding.
use an older path p with obsolete state. More specifically,
due to the global visibility of routing updates from each Principle 2: scalable route update propagation –
individual AS, current inter-domain routing protocol de- Well-structured information flow and high path
signs avoid the use of pro-active, periodic routing updates freshness. SCI-FI provides a scalable architecture where
(like in link-state protocols) to achieve scalability, but use routing updates can be sent proactively to periodically
incremental routing updates which are sent out only after refresh path state, so that each node always maintains a
route changes. However, this incremental manner of rout- fresh (and accurate) network topology on which routing
ing updates compromises route freshness, as the loss (or decisions can be more efficiently made.
malicious drop) of incremental updates concerning a cer- Principle 3: Mutual Controllable path selection
tain path p (such as path withdrawal messages) can pre- – Joint path selection between source and des-
vent other ASes from receiving the update. These ASes tination. SCI-FI greatly increases both the source and
may thus keep the path p with the obsolete state. Con- destination’s ability to affect, select and control the con-
sider the example in Figure 1(b) for illustration, where struction of the routes to and from themselves, while still
the AS PATH {C, M, A, E} is active to reach destination fully respecting intermediate ISPs’ preference and not ex-
E. Suppose that A withdraws E’s prefix, however the posing the remainder of the network to the potential for
malicious AS M intentionally suppresses this withdrawal endpoint-based source routing attacks.
message from C. Consequently, the same AS PATH still
remains active, because B’s withdrawal of the prefix does
not affect the path through M . 4 Architecture Overview
4.1 Hierarchical Decomposition (Sec-
3 Design Principles and Overview tion 5)
SCI-FI is based upon three grounding principles: domain- SCI-FI first divides the Internet into a hierarchy of trust
based isolation, proactive and scalable route update prop- domains, or TDs, as shown in Figure 2, used to provide
agation, and mutually controllable path selection. These the domain-based isolation property. All nodes in the
principles, which we detail below, provide a framework architecture know a set of paths from themselves to a
within which SCI-FI achieves high resilience as a natu- set of exit/entrance nodes from their trust domains, and
ral outcome. Figure 2 illustrates SCI-FI’s domain-based they make these paths available via a lookup service. To
architecture. send data to a node in another TD, the sender selects
an “up-path” from themselves to one of the egress nodes
3.1 Design Principles from their (highest-level) trust domain, and can pick from
among the published paths back to the receiver within
Principle 1: Domain-based isolation – Dividing that domain. In this way, nodes retain complete control
the routing control plane into independent do- over the paths their packets take within their own trust
mains. With properly designed isolation among inde- domains, and the sender is given the freedom to choose
pendent domains, routing in one domain cleanly protects from among those allowed paths. Below, we discuss how
itself from malicious activities and routing churn stem- each of these aspects works.
ming from other domains, benefiting both security and Each TD has a TD Core, a set of specifically designated
scalability while retaining reachability and path diver- Autonomous Domains (ADs) forming a mutually reach-
sity across domains. SCI-FI also lets domains have a able clique that interfaces with other TDs. In the current

3
Internet, the top-tier ISPs would constitute the TD Core.
The architecture defines the AD, or autonomous domain,
as the atomic failure unit (AFU), representing both ISPs Inter TLTD routing
(or transit ADs) and endpoint ADs.

4.2 Routing and Policy Enforcement Intra TD Core routing

Inter-TD routing takes place using human-configured

routes or a path-vector protocol such as BGP. Given the
envisoned small size of TD-level topology (e.g., a few hun-
dred TDs), scalability and security are no longer major
concerns for Inter-TD routing. We envision the effort
to establish a TD to closely mirror that of starting a
certificate authority. The cost involved remains moder-
ately high, so after after a few hundred TDs, the benefit
of creating additional TDs when considering the present sub-TD
amount of ADs will not justify the effort. ADs will em- Top Level Trust Domain
ploy IGP or OSPF for routing within their confines.
Up-path Construction (Section 6) SCI-FI uses Figure 3: A Top-level trust domain. Black nodes are
“up-paths,” a set of valley-free paths, to reach the core. ADs in the TD Core. Arrows indicate customer-provider
The nodes in the TD Core transmit multiple one-hop relationships. Dashed lines indicate peering relationships.
paths to their customer ADs via up-path construction
beacons. These providers then disseminate core reach-
ability information downwards to their customers. The 3. Source nodes apply policy to select an outbound-
endpoint ADs then selects among the k up-paths to the path to the TD Core and a path from the list of
core received from each provider forming k, ideally max- k inbound paths retrieved from the Path Server.
imally disjoint, paths. Finally, the endpoint ADs publish
these paths on the TD’s Path Server, a service located in
the TD Core, queried by local and foreign ADs for rout- 4.3 Forwarding
ing information. For added flexibility, ADs may announce Forwarding remains simple, as this scheme eliminates the
paths to specific ingress/egress points. need for AD-level forwarding tables, making them only
Lookup (Section 7) Name resolution occurs intrin- necessary at the TD level. Forwarding takes place by
sically within the architecture. Each TD will provide simply verifying the specified route in the packet and de-
address servers within the TD Core that resolve human livering the packet on the specified interface.
readable names to addresses and TDs.
Route joining (Section 8) An AD chooses one of 5 Anatomy of a Trust Domain
the up-paths constructed earlier to reach the TD Core
and query the destination TD’s Path Server for the desti- A trust domain (TD) is the fundamental unit of trust
nation’s inbound path list. The source then splices one of in the SCI-FI architecture. TDs are communities of net-
the offered paths onto its own choice of up-path to reach work entities held together by enforceable rules such as
the destination. The destination can simply reverse the contracts, shared legislative and judicial frameworks, or
embedded path or query the source’s path server for al- physical locality. SCI-FI groups entities into these ag-
ternative down-paths to reach the source. Before naively gregates to avoid a non-scalable network architecture in
splicing an up-path to a down-path, the principal will which every entity (ISP, endpoint, or application) must
compare the two paths for common ancestors or peering pairwise evaluate the trustworthiness of every other ad-
neighbors along the path. ministrative entity in the network. Given these aggre-
Policy Decisions SCI-FI reduces policy decisions by gates, the fundamental goal of the architecture is to en-
all stakeholders to three levels: force isolation between TDs while allowing interconnec-
tion. Each trust domain can be considered an indepen-
1. Transit ADs apply their local policy when deciding dent networking plane which can be shielded from the
which paths to permeate downward via the up-path external influence of entities in other trust domains. The
construction beacon. global goal of the architecture is to allow any endpoint to
explicitly specify which of set of these networking planes
2. endpoint ADs apply policy in their selection of k up- it wishes to use and facilitate a connection based on these
paths to publish as inbound to the TD’s Path Server. requirements.

4
 The architecture of a trust domain is shown in Fig- 5.1 Trust Domain Membership
ure 3. Conceptually, it is a contiguous set of ADs, along
with explicitly marked customer-provider relationships. We assume that each trust domain is administered by
A specially designated set of ADs, the top-level ADs or some kind of identifiable organization (a government, an
TD Core ADs, represent the top level of the AD hierar- industry consortium, etc). When a new AD wishes to join
chy: this set contains the entities that perform several a TD, the TD authority must first ensure that it is able
authoritative functions of the trust domain in the various to enforce whatever rules and guidelines that membership
protocols described in this paper. We enforce the con- entails regarding the TD (for example, a country-based
straint that there cannot be any cycles in the customer- TD may require that the corresponding ISPs be regis-
provider graph; furthermore, any AD that does not have tered and headquartered within a given country). Then
a provider in the TD must be a TD Core AD. This en- the TD authority determines the topological relationship
sures that every node in the network has a strictly up- of the joining ISP with respect to the current TD. To
stream path (i.e., a path that traverses edges only in the join an existing trust domain, the new AD should either
customer-to-provider direction) that leads to one of the 1) have one provider already inside this TD, or 2) be
TD Core ADs. The set of TD Core ADs must be con- capable of being a core AD in this TD. Specifically, to
nected and mutually reachable in the AD-graph. Since meet the second requirement, the new AD needs to be
most TD Core ADs in the current Internet are mutually able to directly reach other core ADs by communicating
peering, this topology is expected to be extremely simple. with only other core ADs, as well as satisfy a subject as-
Mutual reachability can be either implemented as a set sessment of being a sufficiently well-connected AD: this
of human-configured primary and backup routes for TDs requirement is similar to customer / traffic requirements
with a small set of TD Core ADs, or a separate path- for peering.
vector protocol (similar to S-BGP) can be used for larger When an AD establishes a new service connection with
numbers of TD Core ADs. an AD in a different trust domain, it needs to inherit
one or more of the TD associations of the provider in
order to access the relevant sets of upstream paths of that
A trust domain can be a top-level trust domain provider. The exact TD assignment is dependent on the
(TLTD), or a sub-TD. A sub-TD is wholly contained terms of the service and contingent on whether the child
within a TLTD and may contain other sub-TDs. A top- AD can satisfy the conditions of joining the new TDs.
level TD is not contained in any other TDs (although its
member set may overlap partially with other TDs). We
5.2 Subsidiary Trust Domains
assume that there will be relatively few top-level TDs in
the world (between 10 to 100), with each TD correspond- A top-level TD may contain subsidiary trust domains
ing to a large, globally identifiable real-world group (such (“sub-TDs”). A sub TD is depicted inside the main top-
as a country, or a well-known international organization). level TD of Figure 3. The purpose of a sub-TD is to al-
low finer-grained trust domain selection (for example, the
armed forces of a country may operate a sub-TD within
The TD Core ADs in the top-level TDs facilitate inter- its own country’s TD, to support a higher level of assur-
connection between top-level TDs using the Inter-TLTD ance than civilian ISPs). A sub-TD is internally struc-
routing protocol. This protocol is essentially a path- tured similarly to a top-level TD, with its own TD Core
vector protocol identical to S-BGP, except at the gran- ADs which are mutually-connected, its own route servers,
ularity of TLTDs instead of ADs; since this topology is and so on. The only difference between a sub-TD and a
extremely small and densely connected (the majority of top-level TD is that the TD Core ADs of a sub-TD does
routes should not need to traverse more than 2 TDs), not take part in the the inter-top-level TD (inter TLTD)
most of the routes are static and can be directly con- protocol.
figured. When automatic route discovery is needed we
assume that TD-level routing policy (e.g., which TD to
use to reach a distant TD) is agreed-upon among the TD 5.3 Benefits of Using Trust Domains
Core ADs beforehand. To facilitate the Inter-TLTD rout-
We end this section with a list of intrinsic benefits of
ing protocol, the TD Core ADs engage in a protocol to
building the inter-domain routing architecture based on
discover their mutual interfaces to other TLTDs in a man-
the notion of TDs.
ner similar to IGP; since there are only a few of these TD
Security against malicious attacks. Due to the
Core ADs per TLTD, each TD Core AD can simply keep
a table of what TLTDs are reachable from each of its fel- strong isolation and control that TDs provide, multiple
low TD Core ADs. Inter-TD packets (from ADs in one long-standing attacks are naturally eliminated in SCI-FI.
TD to ADs in another TD) are tunnelled across this pro- For example, the worm-hole attacks at large scale are
tocol via encrypting at the TD Core AD gateways such no longer possible in SCI-FI, because routing paths are
that forwarding routers from third-party TDs are unable constructed and authenticated within each domain sepa-
to inspect the packets that belong to the endpoint TDs. rately to provide a natural defense against cross-domain

5
 notation meaning
worm-hole paths. In SCI-FI, two colluding ADs can P-C link Pij AD i → AD j AD i is AD j ’s provider
only create worm-hole up-paths within the same domain, peering Qij AD i ↔ AD j AD i and AD j are peers
t
which can only incur limited damage given the “shallow- timestamping r
AD i −→AD j link is up at time tr
TD
tree” structure of the Internet [18]. link scoping AD i −→AD j link is valid in TD
AD cert Certi∈T D TD certifies AD i is in TD
Resilience against human misconfiguration. The sub-TD cert CertT D1⊆T D TD certifies TD1 is its sub-TD
TD Core cert CertT D→i TD certifies AD i is TD Core AD
intrinsic isolation provided by the division of TDs peering cert Signi(Qi,j ) AD i certifies a peer AD j
achieves an in-depth resilience against human misconfig- up-path cert Signi(Pj ) AD i certifies one of its up-path Pj
uration, which proves to be the most popular reason for
current routing system outages [19]. First, designing a Table 1: Notation.
robust inter-domain routing architecture based on TDs is
conceptually simple and convenient, which can help re-
duce human configuration errors compared to “ad hoc” to TDs and their respective public keys.
engineering hacks used in the current practice. Second,
even if a misconfiguration happens, the damage is only Up-path definition. The fundamental unit of rout-
confined within that TD because a routing update is only ing in the SCI-FI architecture is the upstream path, or
propagated within the local TD. “up-path” for short. To support multi-path routing, the
protocol enables all ADs to be supplied with multiple dis-
Elimination of a single point of trust/failure. tinct policy-compliant AD-level paths to reach a TD Core.
The existence of multiple TDs eliminates the need for To keep route lookup overhead practical, we restrict the
a single authority of the entire Internet, which causes de- number of paths that each endpoint AD maintains in its
ployment issues and a single point of failure. reachability record to at most k per TD. In facilitating the
construction of these k up-paths, the upstream ADs are
not required to exhaustively enumerate all possible paths
6 up-path Construction to all possible TD Core, but simply provide a sufficient
number of alternatives.
In this section we describe how each AD learns of its
AS-level paths to the TD Core ADs through periodic up- Overview of up-path construction. The path
path construction beacons containing routing information. discovery process starts at the TD Core in each TD where
The process of obtaining this information consists first of each TD Core AD initiates an up-path construction
trust bootstrap to establish the identity of the TD core; beacon in each time period. Each AD passes along
second, of the beaconing process itself; and third, of us- an up-path construction beacon to each customer after
ing the beacon-derived information to construct one or receiving such a message from its provider and appending
more up-paths. Also, SCI-FI supports routing at the necessary information. Specifically, a up-path construc-
ingress/egress point level to enable finer-grain routing tion beacon has three components: U = {P, S, M }. P
control. Finally, to reach nodes outside the TD, SCI-FI contains an up-path in a containing TD. Each link on
provides a mechanism to traverse TD boundaries. this up-path is timestamped and scoped to a TD. S
In the next section we describe how to lookup destina- is a cryptographic authenticator signing the path and
tion AD’s reachability information. Combining these two certifying every on-path AD’ TD membership. Finally,
parts enables paths to be discovered between endpoints. M represents a capability token enabling efficient for-
warding control as proposed in prior work [20, 21, 22].
6.1 Management and Trust Bootstrap
Fig. 4 illustrates the high-level idea of up-path con-
We assume that the TD Core members of each TD act struction for a simple AD topology where all ADs belong
as a central authority that administers the TD. This au- to the same TD and there is no peering links. AD1 is
thority may in fact be implemented using a distributed the TD Core initiates periodic up-path construction bea-
functionality; for simplicity we assume it is a single en- cons. AD1 sends U12 and U13 to its customer AD2 and
tity. Each top-level trust domain is associated with a AD3 , respectively. After receiving U13 , AD3 appends
fixed human readable identifier as well as a well-known information regarding the link to its customer AD4 to
public/private keypair. Due to the high level of visibility construct U134 . AD4 , which receives two up-path con-
of the top-level trust domains, we assume that bootstrap- struction beacon from its upstream providers, can decide
ping the well-known TD authority public key onto the which one(s) to export to the downstream AD based on
relevant principals (specifically, the members of the TD the routing policy. In the remainder of this section, we
as well as other top-level TD authorities) is secure. For will extent our up-path construction protocol to support
example, the public key could be passed on by trusted peering links, fine-grain routing based on ingress/egress
service providers. The TD authority then operates a PKI points, and links that cross TD boundaries. Table 1 sum-
CA for the member ADs of the TD, signing certificates of marizes the notation used to express the detailed con-
membership binding ISP identification and AD numbers struction of up-path construction beacon.

6
 1
t ,T D
P13 = AD1 1−→ AD3
U12 U13 = S13 = CertT D→1 kCert3∈T D kSign1(P13 ) stream providers, it can then disseminate them to subse-
M13 = MACK1 (P13 kT D)
2 3
t ,T D
U124 P134 = P13 kAD2 3−→ AD4
4 U134 =
U1245 and/or S134 = S13 kCert4∈T D kSign3(P134 )
U1345 M134 = M13 kMACK3 (P134 kM13 )
5 To identify more shortcuts (as described below), each
Figure 4: An example of up-path construction beacon as follows:
dissemination. Arrows indicate the flow of up-path con-
struction beacon. All ADs are in the same TD. U13 =
{P13 , S13 , M13 } and U134 = {P134 , S134 , M134 }.
6.2 Constructing up-path in one TD
The following protocol is used to construct the up-paths
for each AD. We first assume every provider AD in the
following protocol description is part of the trust domain
T D. We discuss extensions to multiple TDs later.
The protocol starts with the top level ADs. Specifically,
those ADs that do not have any upstream providers that
are also in the trust domain. Each top level AD ADr
sends a message to each of its customer ADs (ADi ) as
(Pi , Si , Mi ) as follows:
tr ,T D
Pi = ADr −→ ADi
Si = CertT D→r kCerti∈T D kSignr(Pi )
Mi = MACKr (Pi kT D)
The path information portion of the message, Pi in-
dicates a one-hop path from ADt to ADi that is times-
tamped with (internal) time tr which makes it valid for a
time depending on the AD’s internal policy. The edge is
also labeled with a trust domain scoping term, T D, indi-
cating in which trust domain it is valid for. The path au-
thentication information Si contains a certificate (signed
by the TD authority) authenticating ADr as a TD Core
AD in the trust domain T D, as well as a membership
certificate authenticating ADi as a member of T D, and
a signature by ADr on the path information. The mes-
sage authentication code (MAC) portion Mi is computed
over the message using a secret key Kr known only to
ADr , and is essentially a data plane capability used to re-
mind ADr of its own decision that Pi is an approved path
when used to carry data packets of the trust domain T D.
Note that since these capability MACs are only verified
by the issuing AD (ADr ) there is no requirement for time
synchronization with other ADs as long as the time syn-
chronization within the routers of each AD are sufficient
to enforce its individual policy regarding route announce-
ment timeouts. Since all IDs are self-certifying, any third
party can extract the relevant public key of ADr from
the ID of ADr contained in Mi , and can thus verify the
signature.
As an intermediate AD receives paths from its up-
quent downstream customers in the natural way. Suppose
an intermediate AD (ADi ) receives a set of paths from its
upstream providers. It checks the signatures on each of
the paths, and discards any ill-formed or unauthenticated
paths with bad signatures.
pair of peering ADs (h, i) also exchange a peering certifier
th
Qh,i = ADh ↔AD i
Sh,i = Signh(Qh,i )
Mh,i = MACKh (Qh,i )
Note that peering certifiers are unscoped by TD since the
TD membership of the AD on each end of the peering edge
should be separately certified on the path authenticator.
For each downstream AD (ADj ), the parent AD (ADi )
then chooses a (preferably maximally disjoint) path set
of m paths P1 , . . . , Pm , where m ≤ k. Each of these
paths necessarily terminates at the parent AD, ADi .
For each path Pp with path authentication information
Sp , ADi considers the set of peering links that it will
support for downstream customers from ADj and at-
taches this information into the path. It then appends
ADj onto the end of the path, and computes new au-
thentication information resulting in the message tuple
Up,j = {Pp,j , Sp,j , Mp,j } as below (this example shows
only one peering edge (h, i), but additional peering edges
can be appended as needed):
ti ,T D
Pp,j = Pp kQh,i −→ ADj
Sp,j = Sp kSh,i kCertj∈T D kSigni(Pp,j kSp kSh,i )
Mp,j = Mp kMh,i kMACKi (Pp,j kMp kMh,i )
This process continues until each AD has obtained a
set of paths terminating at itself, where each edge in the
path is authenticated by each parent AD. Each of these
authenticated edges essentially represents a network ca-
pability given from a parent provider to a customer; when
these routes are used, the parent AD will check the corre-
sponding self-MAC to ensure that the provided path truly
corresponds to a path that it supports for that customer.
As an example, Fig. 4 shows the up-path construc-
tion beacon that is delivered to AS4 for the path
AD1 , AD3 , AD4 . It can be seen that the respective sig-
natures and MACs are constructed in an onion fashion,
with each succeeding authentication tag authenticating
not just the newly extended path, but also all the previ-
ous authentication steps involved in creating that path.
Once an endpoint AD receives the up-path construc-
tion beacon from its providers it selects up to k up-paths
and signs the entire set of k paths and their authenti-
cators and uploads it to a lookup server provided by the
trust domain to enable reachability. To send a packet, the
7
source AD queries the Path Server server to find the k-
paths associated with the destination AD and thus splice
together a route. This process is described in Section 7.
6.3 Managing multiple ingress/egress
points
Often, routing at the AD-level granularity is insufficient.
For example, consider a large carrier with continental
reach but only a single AD number. A route passing
through the AD could be entirely local, or it could cross
the entire continent. To distinguish these cases, BGP
supports an attribute called the “mult-exit discrimina-
tor” or MED, which essentially allows an AD to annotate
its path advertisements with which exits are supported.
Since the up-path construction is essentially also a
path-vector algorithm, the concept of MED can be seam-
lessly integrated into the protocol. When an AD wishes
to constrain an up-path to a certain ingress/egress point,
it can simply annotate the path announcement with this
fact (e.g., ignoring peering edges):
ti ,T D
Pp,j = Pp −→ ADj : MEDx
Sp,j = Sp kSigni(Pp,j kSp )
Mi = Mp kMACKi (Pp,j kMp )
This embedding of the MED into the path description
prevents a downstream customer from selecting routes
that violate the ingress/egress semantics of the provider
AD. The use of this field can also be extended to indi-
cate policies regarding how the provider wishes to sup-
port transit between two customers. For example, if a
provider wishes to restrict a path to only some subset of
its descendants, it could annotate this in a similar way to
the MED. For brevity, we do not describe policy annota-
tions in detail in this paper.
6.4 Traversing TD Boundaries
The protocol above guarantees that a customer AD can
discover a route from a provider as long as there exists a
common trust domain to which both of the ADs belong.
However, the protocol does not describe how a customer
AD (ADj ) in some trust domain (TD1) could use a route
Pp from a provider (ADj ) in a different trust domain
(TD2) when TD1 and TD2 are in different TDs.
In some cases two ADs may still want to communicate
even though they are in different TDs. SCI-FI support
the use of routes that cross TD boundaries at the cost of
isolation by expanding the scope of a path to a “univer-
sal” trust domain, TD*. TD* is a virtual domain that
contains all TDs. By introducing the TD* domain, every
pair of ADs now have at least one common containing
domain. For example, consider the case that the next
largest containing TD of TD1 and TD2 is TD*. In this
case, the provider ADi will re-scope the advertisement
from TD2 to TD* by changing the context of the edge
8
i → j to TD*. Subsequently this route can be identified
as a valid TD* route but not a valid TD2 route. By using
the TD* domain, we can still support use of the route at
a lower assurance level. Since TD* is a virtual domain
without authorities or servers, routes for TD* are hosted
at the route servers of the originating TD.
7 Lookup
Once an endpoint D has selected its k up-paths, D must
publish this reachability information in a reliable man-
ner such that source endpoints may determine suitable
routes to D. To support this lookup, each TD Core runs
replicated Address Server(s) and Path Server(s) that are
reachable via a well-known address.
The following steps take place in the lookup process: we
start with a source endpoint wishing to reach a destina-
tion endpoint, which is identified using a human-readable
label and a TD identifier, defaulting to the local TD. The
source endpoint issues a query to the local Address Server
for resolution of the label. The TD’s Address Server
resolves this query by returning an identifier indicating
which AD(s) serves an endpoint associated with the la-
bel, as well as which sub-TDs contain this AD ID. The
source can use this TD-membership information to query
the Path Server in an appropriate TD for the k-up-paths
of the destination. After discovering the set of up-paths,
the source constructs a route from itself to the destination
endpoint by selecting a segment of one of the k paths.
We assume that knowledge of the identity of the TD
Core implies possession of the public key of that TD Core;
this is similar to the assumption of, e.g., browsers having
ICANN’s root public key for DNSSEC. Also, to certify the
mapping between the endpoint identifier and the address,
each TD Core effectively maintains its own autonomous
endpoint identifier space. Hence, an endpoint identifier
can be an unambiguous human readable name (as long
as it includes at least one TD identifier). For exam-
ple, an identifier for the US-based car manufacturer Ford
may be “Ford” qualified with the “USA” TD. Naturally,
this system would also support cryptographic identifiers
made of non-human-readable bitstrings, such as AIP [23].
For simplicity, we assume that human-readable names are
used in the rest of this paper.
7.1 Address Resolution Service
Name resolution in SCI-FI occurs in accordance with
the same foundational principles of the routing design,
namely the isolation of influence to the TDs involved.
The architecture devolves control to the endhosts, allow-
ing them to scope name resolution. Furthermore, the
design settles disputes by resolving non-TD specified ad-
dresses at the local domain, where presumably an enforce-
able dispute resolution process exists.
Every TD will provide its own internal Address Server
within the TD Core, accessible at a default address, which
will resolve the name locally if possible, or query the ap-
propriate foreign Address Server should the user specify a
different TD. Should the local Address Server not contain
the name, the server will query the other TD’s Address
Server on the user’s behalf. While SCI-FI remains ag-
nostic to the exact scheme for human readable naming,
the example of DNS provides the most accessible exam-
ple. Consider the domains ABC.us and ABC.cn, residing
within the US and CN TD’s respectively. A user within
the US TD will query the local Address Server and receive
address information to ABC within the US TD. Should
the user in the US TD request ABC.cn, the US Address
Server queries the CN Address Server, and returns this
information to the user.
The namespace can be flat, or hierarchical. For now,
we can assume that the service is structured similarly to
DNS: there is one canonical root server associated with
each trust domain, which can delegate the lookup to a
number of sub-servers in possible other sub trust domains,
until a query is resolved.
To look up a path, the source queries the local Address
Server with the destination identifier. The destination
identifier consists of a human-readable label, such as a
DNS name, and a TD identifier, which defaults to that of
the local TD. The Address Server returns an identifier in-
dicating which AD(s) serves an endpoint associated with
the label, along with which sub-TDs contain the AD ID.
If the local Address Server cannot resolve an identifier lo-
cally, the Address Server queries the other TDs’ Address
Servers.
The name resolution service takes as an input a human
readable name: NE (e.g., “Ford”) and a TD Identifier
C (e.g., “USA”), and outputs a list of (self-certifying)
AD numbers and endpoint identifiers: each result is an
AID:EID pair constructed in a way similar to that of AIP.
Optionally associated with each AID:EID pair is a hier-
archical nesting of trust domains (e.g., local, regional,
and continental trust domains) that can be used to dele-
gate the reachability functionality to sub-trust domains.
Specifically, query on NE at TD C should produce the
following record:
NE in TD C resolves to:
AID1 : EID1
AID1 ∈ T D1,1 ⊆ T D1,2 ⊆ T LDC
AID1 : EID2
AID2 : EID3 AID2 ∈ T LDC
Table 2: Example of a lookup record for name NE in TD
C the query. For top-level scoped reachability queries, the
In the example of Table 2, the lookup of NE under TD
C returned three records: two endpoints EID1 , EID2 in
the same AD (AID1 ) and another in a second AD, AID2 .
The trust domain memberships of each AD are indicated:
for example, for AID1 , it is contained in T D1,1 which is
a sub-domain of T D1,2 which is a subdomain of the top
level domain T LDC . For AID2 , it is simply indicated as
a member of the top level domain T LDC .
9
The top level TD C signs a certificate authenticating
the correctness of this name resolution lookup. Each self-
certifying AD identifier (or AID) comes with a co-signed
certificate attesting to the membership of the AID in each
of its respective trust domains, in particular C. For each
AID:EID pair there is also a co-signed certificate indicat-
ing that the EID is part of the AID.
The name resolution query is routed using the identifier
of the TD C. A querying source endpoint sends this query
to the targeted top level TD C via one of its up-paths P ,
into the top level routing infrastructure which sends it
to the correct ingress point at C. TLTD C then resolves
the query (possibly by querying sub-TDs) and returns the
response via the originating up-path P .
7.2 k-Path Resolution
At this point the endpoint is in possession of the AID:EID
of the label that was looked up, as well as possibly a
hierarchy of nested trust domains that contain this AID,
but not the actual route itself. The source now issues
a route lookup query to the respective trust domains as
appropriate (e.g., if the AID is contained in TD1 which
is contained in TD2, and the source knows how to reach
the Path Servers of TD1, it can contact TD1 directly). In
the following description we assume that the source has
no such advance information and can only reach the top
level TD.
The destination AD actively uploads its k up paths
on any or all of its containing TD’s Path Servers. The
route resolution query from the source first goes to the
top level TD ingress point, which may then query one of
the top level Path Servers to see if the route has been
uploaded at top level. If not, or if the top level trust
domain prefers not to resolve individual AD routes, it may
also delegate the lookup based on the TD containment
information provided in the lookup query. Eventually, a
trust domain is found that contains the destination AD
and whose Path Servers have a fresh copy of the k up
paths of the AID.
Trust-scoped resolution. A trust scoped query en-
forces that a given route computation should only involve
(and be restricted to paths using) ADs within a specific
set of trust domains. For example, if a query is scoped
to within top level domain C, any AD that is not either
explicitly specified by the querier, or in C, should not be
able to directly affect the communication or the results of
implementation is straightforward. The querier sets a flag
in its query indicating that this is a trust-scoped query
restricted to the top level trust domain C. Then, when
a Path Server is queried for routes, it simply withholds
any paths that are not scoped for C. If the Path Server
needs to follow a delegation path (e.g., querying the Path
Server of a sub-TD), the Path Server ensures that the
delegation path never exits C (in terms of forwarders or
communication principals). This effectively restricts the
set of participants in the route computation strictly to
members of C. A label lookup can be scoped to a top
level domain in a similar way.
Subtleties arise if a more narrow scoping is desired. For
example, suppose a source issues a route query (AID1 :
EID1 )(AID1 ∈ T D1,1 ⊆ T D1,2 ⊆ T LDC ), with the trust
scoping constraint of T D1,2 (where T D1,2 ⊆ T LDC ).
Since T D1,2 may not be a top level TD, and may not
participate in the inter-TD top level routing layer, this
implies that the route query is expected to traverse T LDC
but that the ADs in T LDC (and, indeed the authority of
T LDC itself) may not be trusted. In such a situation,
the source is required to designate a particular explicitly
source-selected route to reach T D1,2 . The ADs in this
route are allowed to drop the packet if it violates their
routing policy but are expected to strictly follow the se-
mantics of the routing otherwise. It is the responsibility of
the source to discover a trusted route to reach the trusted
TD T D1,2 ; this can involve a separate route query (possi-
bly, scoped to a group which contains only the high-level
members of the top level domain T LDC ), or the route
can be hand-configured based on source preferences.
In-bound traffic control. Another way for the pol-
icy to be reflected is via in-bound traffic control. A des-
tination AD can attach distinct sets of k-path upgraphs
depending on the identity of the source endpoints that are
performing the query. For example, if a query is originat-
ing from outside the local TD, the destination AD can
instruct the Path Server to only serve up a set of paths
that pass through a set of specific high-security gateway
ADs; whereas if the query originates inside the TD then
a more general and efficient set of paths can be served up.
8 Route Joining
Section 6 described the process of route construction
(where a destination AD determines its upstream topol-
ogy) and Section 7 described route lookup (where a source
AD discovers the k up-paths of the destination AD). It
remains to show how this information can be used to con-
struct a working end-to-end route. The joining algorithm
occurs at the source endpoint. It can be implemented ei-
ther on the network stack of the endpoint device, or on
the endpoint routers of the source AD. It takes as inputs
the k up-paths of both the source and destination hosts.
Each up-graph is a list of k path vectors, where each path
vector is in the following format:
hAkQ(A) → BkQ(B) → CkQ(C) → . . .i
where A, B, C, . . . are the nodes (in order) in the path,
and Q(A) is the set of peering neighbors of node A, that
A attached onto the path. The use of Q(A) is to enable
the source and destination to find not only joining points
at a common ancestor, but also join points at a common
peering link. For example in Figure 5, without consider-
ing the peering links, node A and B can only find a route
10
Root
E
C D
A B
Figure 5: Short cut example. The link between C and D
is a peering link; other links are customer-provider links.
A, C, E, D, B. When also comparing the peering nodes
at the up-paths, A and B are able to find a shorter path
A, C, D, B.
The source endpoint needs to find a common ances-
tor provider or a common peering link to splice together
the up-path of the source and the destination AD. If the
source and destination ADs are in distinct trust domains,
then the route may need to traverse top-level TDs via the
inter-TLTD protocol. Similarly, if source and destination
are in the same trust domain but have distinct TD Cores,
then the message will need to traverse the TD Cores in
the same TD via the inter-TD Core routing protocol. To
reflect this, a common ancestor C is attached as the par-
ent of all top-level ADs in both path sets. This ensures
that a common ancestor can always be found.
Finding the common join point can be accomplished via
a number of methods. Since we anticipate that the paths
will be reasonably short and k is also quite small (less than
10), a simple way is to hash the IDs of each provider and
peering link of the destination AD up-paths into a hash
table and look up the providers and peering links of the
source AD. This takes time and space proportional to the
total number of provider ADs and peering links named in
both source and destination.
Once a common transition point X (either a common
ancestor or a common peering edge) is found between an
up-path PA of the destination AD A and an up-path PB
of the source AD B, the endpoints are ready to initiate
communication. The source endpoint attaches into the
packet PA and PB , along with their MAC authentication
information (MA and MB ) as constructed in Section 6.
The header also contains the transition point X that was
selected by the source endpoint. It now contains enough
information to be routed to the destination. On-route,
routers inspect the header to determine the next AD hop,
as well as check the MACs that locally authenticate that
this is a well-constructed, legitimate route. When the
packet has reached the destination, the destination can
reverse the paths to construct a symmetric return path,
thus facilitating two-way communication. During the life-
time of the connection, the source endpoint can moni-
tor path quality and switch to any of the other k2 com-
bination of alternative path choices to improve latency, ISP attempting to implement a highly reliable service to
throughput, or drop rates. other important ISPs. Currently, without setting up a
dedicated sub-network for this transport, it cannot do this
in existing routing protocols. We would like to facilitate
9 Security Properties this naturally within inter-domain routing, by providing
route control and selection properties at the AD level.
Rather than design our architecture with specific coun- In existing interdomain routing protocols, a destination
termeasures against known attacks, we have designed it AD has essentially no control over its route: it must sim-
using sound principles of isolation, control and scalabil- ply advertise its prefix to its providers and hope for the
ity such that security follows as a natural intrinsic prop- best. The source AD has very limited control since, if
erty rather than as an external property added-on by a it is multi-homed, it can pick any one of its providers
separate subprotocol. This section discusses some of the to serve the route. If it is not multi-homed (as many
natural resilience afforded by our architecture in contrast stub ADs are) then it likewise has no route choice once it
with the current Internet infrastructure. has selected its provider. In the SCI-FI architecture, in
contrast, both endpoints ADs get to selection a set of k
Isolation: enabling attack localization. As dis- well-defined paths from the path set provided from their
cussed in Section 2, the primary weakness of current in- respective providers. This choice is made explicitly, with
terdomain routing protocols (even with their semantics full knowledge of the exact identities supplying the path
perfectly secured) is that the system is vulnerable to var- and the full authentication information of each path is
ious forms of routing plane attack to any on-path ad- provided. A source AD gets both these sets to choose
versary, i.e., any adversary that is, or could make itself from, yielding potentially up to k2 end to end AD-level
be, on any path between the source and the destination. paths. A destination AD can in fact select different sets
Since the routing announcements in those routing proto- of k paths to serve up to different source endpoints; for
cols are globally disseminated without isolation and con- example, the destination AD can approve a separate set
trol, this includes every AD in the world to a varying of high-assurance paths for trusted entities; this can be
degree; closer ADs have increased attack power. In con- provided even if the TD route servers are untrusted by
trast, SCI-FI provides strict isolation and control prop- encrypting the route record using a group key. Further-
erties. Since customers never “announce” any routing more, these route sets are all separated by trust domain,
control messages to providers, influence over route con- with each domain maintaining a different set of k paths.
struction flows strictly downstream: this means that any An AD that is in more than one TD can thus not only
malicious provider can only attack routes belonging to its switch paths but also change the routing context to a
own customers. Furthermore, since routing control is di- different TD.
vided by trust domain, a malicious AD can only attack
routes which have at least one endpoint inside its trust Control: facilitating resource control. In terms
domain. The attacker is limited to a limited capability of resource management, TD separation can also be used
of falsifying upstream routes within the same domain to to provide a form of resilience against resource consump-
downstream customer ADs of the malicious AD: it can- tion attacks and DoS. For example, routers could allo-
not, for example, attempt to forge a false route lookup cate independent bandwidth to each TD. An adversary
result from another TD. Finally, notice that the target of that does not control any malicious routers in a given
these attacks have to be downstream endpoint ADs, who TD X will never be able to eavesdrop on route adver-
are the final approvers of the route computation (each tisements belonging in X. Hence, the adversary cannot
endpoint must explicitly select k up paths, sign the selec- consume router resources dedicated to TD X unless it
tion and upload it to a route server). Because the target is specifically targets an endpoint inside X. However, since
actively involved in the final route approval stage, attack- this endpoint is legitimate, it can shut down the adver-
ing the up-path computation cannot be done stealthily: sary’s access to its routes in a number of ways: it could
the target of the attack always has a chance to examine blacklist the malicious AD at the route server, causing
the forged route and always has to approve it explicitly. it to be unable to receive new capabilities to send pack-
Compare this, for example, with the case of path compu- ets once its current capabilities time out; or if the end-
tation in BGP, where a destination has no control over, point determines that it is under DoS it could ensure
and is often unaware of what routes to itself are eventu- that all unauthorized traffic passes through a dedicated
ally adopted by the rest of the Internet. lower-priority AD by serving up alternative paths to well-
established trusted flows. Existing interdomain routing
Control: providing resilience and flexibility. An- protocols have only one single routing plane and is thus
other advantage of the architecture is resilience and flexi- unable to perform this kind of isolation.
bility. When an AD is unable to determine or control the
route that it uses to another AD, it is unable to facilitate Scalability: preventing stale routes. SCI-FI also
a well-quantified level of trustworthiness or reliability for supports a much more flexible and fine-grained ability to
its network service provision. For example, consider an manage route updating. In existing interdomain routing

11
 Isolation
BGP no isolation; can be attacked by any AD on any path
SCI-FI isolation to restrict attack scope; can only be given
false route data by upstream providers in the same TD
Path control
BGP Dest: no path choice
Source: pick one out of d paths
SCI-FI Source/Dest: pick k out of kd path segments
Dest: can enforce different paths for different traffic classes
Source: pick k2 ways to combine segments into path
Resource Control
BGP Unsupported
SCI-FI Reliably segregate traffic by TD on data & control planes
Scalability
BGP attacker can drop route withdrawal to keep stale routes
SCI-FI fresh routes due to scalably frequent route update
Table 3: SCI-FI Security Advantages. d is number of
providers for multihome source/dest.
systems, the only mechanism for route changes is to in-
crementally push out another routing update or a route
withdrawal. Because there is no route control from the
destination, these updates may or may not propagate cor-
rectly to other ADs. Hence, stale (invalid) routes may
remain in routing tables for a long time. Furthermore,
attempting to secure these explicit updates is problem-
atic. An attacker could re-order or re-inject route update
messages causing invalid and inconsistent routes to prop-
agate in the network. Path-vector based route announce-
ments cannot have short timeouts, since a path-vector
update requires a destination AD to push its announce-
ment to the entire network, and the protocol is not scal-
able if every AD is performing this broadcast at a high
rate like link-state routing, since this would cause O(n2 )
communication overhead per update (since it involves all-
to-all communication). In SCI-FI on the other hand, all
routes come essentially directly from the destination AD
via the route servers of the TD. There is no global dis-
tributed consistency issue since the route servers are cen-
trally administered. Since all route discovery occurs in an
upstream-to-downstream direction, frequent updates can
be done highly scalably (O(n) communication per up-
date) in a coordinated fashion. Hence, published routes
can have very short timeouts and be updated frequently
and securely.
These observations are not exhaustive; they are simply
examples of a number of BGP flaws that are naturally
prevented through good design supporting well-founded
isolation and control. Note that none of these useful prop-
erties required an additional add-on protocol to provide
functional security, and the validity of each property is
quite clear without requiring extensive cryptographic re-
duction proofs. A summary of the properties discussed
in this section is in Table 3.
10 Evaluation
Due to the infeasibility of evaluating a completely new
architecture on the current Internet, we have constructed
12
an AD topology based on real-world datasets to evaluate
the effectiveness of SCI-FI. Specifically, we simulate SCI-
FI on a measured Internet AD topology from the CAIDA
dataset to evaluate the routing efficiency, security, and
expressiveness.
10.1 Evaluation Methodology
We evaluate our protocol on the measured AD-level topol-
ogy annotated with the business relationships obtained
from CAIDA dataset.1
Trust domains. Given the measured AD topology, we
virtually group the ADs into several trust domains and as-
sign some of the ADs as the TD Core ADs in order to sim-
ulate our trust-domain-based routing. In this proof-of-
concept evaluation, we consider a two-level trust domain
hierarchy in which the lower level consists of five local and
non-overlapping trust domains, and the upper level is the
T D∗ domain containing all the lower level TDs. Each of
these five local TDs associates with one Regional Internet
Registries (RIRs), which are the regional organizations al-
locating AD numbers. In other words, ADs registered to
the same RIR belong to the same trust domain. Such a
division reflects the geographical and administrative re-
lationships to some extent. Table 4 summarizes the size
of each of the trust domains. The TD Core ADs are de-
fined as top-level ADs that have no providers themselves
in their respective trust domain.
BGP Routing. When simulating BGP (and S-BGP)
routing,2 we assume that in benign cases ADs respect
and make routing decisions based on the business rela-
tionships with their neighbors, and then use path length
as the tie-breaking factor. We also assume that the TD
Core ADs form a clique, and thus the length of any in-
ter TD Core routing path is 1. This is accomplished by
adding a peering link (if it does not yet exist) between
every pair of top-level ADs.
Finding k up-paths. In practice, each AD running
SCI-FI can have a different set of policies in determining
which k paths to export, and the infrastructure can vary
the value of k as well as the algorithm for finding the
(presumably disjoint) k paths. However, the optimiza-
tion of the export policies, the k value, and the finding-
disjoint-path algorithm are outside the scope of this pa-
per. Instead, for the purpose of simulation, we implement
a simple k-path discovery algorithm that takes a source
AD, the complete AD-level topology, and a Trust Domain
as inputs, and yields an up-path containing k maximally
edge disjoint paths to the TD Core ADs in the specified
Trust Domain. The existing algorithms of “finding k dis-
joint paths” focus on minimizing the total cost of the k
paths. Such algorithms are inapplicable in the context of
1 CAIDA. http://as-rank.caida.org/data/
2 The reason we compare with BGP is because it is the current de
facto interdomain protocol and the CAIDA dataset directly reflects
BGP paths.
 Trust Domain
African Network Information Centre (AfriNIC)
American Registry for Internet Numbers (ARIN)
Asia-Pacific Network Information Centre (APNIC)
Latin America and Caribbean Network Information Centre (LACNIC)
RIPE NCC
Table 4: Results of trust domain formation.
8000
in TD*
7000 in local TD
6000
number of ASes
5000
4000
3000
2000
1000
0
1 10
number of available paths (logarithmic)
Figure 6: Measurement results of AD-level end-to-end
path diversity.
finding routing paths in SCI-FI because the probability of
using a backup path may be much lower than using a high
priority path. For example, one may prefer a set of paths
with cost {2,10} to another set with cost {6,6} despite
that the total costs are the same. Therefore, rather than
minimizing the total cost, our k-up-path discovery algo-
rithm selects the disjoint paths using an iterative, greedy
algorithm. At step i, the greedy algorithm 1) finds the
current shortest path as the ith maximally disjoint path,
and 2) increases the weight of all the edges on the ith path
such that these edges become less preferred in the next
iteration. Fig. 6 shows the distribution of the number of
available paths. More than 90% and 80% of ADs have
fewer than 10 available paths in the sub TDs and TD*,
respectively.
10.2 SCI-FI Efficiency: Route Stretch
We evaluate the efficiency of SCI-FI based on its route
stretch, which is defined as the ratio of the average path
length in SCI-FI to the average path length in BGP. Our
simulation takes 1000 random pairs of source and desti-
nation ADs. For each pair, we measure the length of the
BGP route and the SCI-FI route between the source and
the destination. The result shows that the average BGP
path length is 3.9 hops and 4.72 hops in SCI-FI when
k = 1. Our result shows that routing in SCI-FI only adds
a small amount of overhead in terms of path length even
without considering the shortcut paths that do not go
through the TD Core ADs.
number of ADs number of top-level ADs
613 39
21619 38
6039 29
1912 60
19569 34
10.3 Expressiveness
As explained in Section V, with the ability to re-scope to
the TD* universal domain, SCI-FI can express all the
valley-free BGP paths, because the TD* domain con-
tains all the ADs and thus routing in the TD* domain
is essentially equivalent to routing in the Internet us-
ing BGP. Specifically, we can show that every policy-
compliant BGP path between a pair of source and des-
tination ADs is always contained in the union of their
100 full up-paths (i.e., k → ∞) in the TD* domain. We con-
sider the full up-paths because the selection of k (and also
which k paths) is a policy decision, and designing an al-
gorithm to optimize such a selection is outside the scope
of our paper.
However, re-scoping to the TD* domain sometimes is
undesired because it achieves only a minimal security as-
surance. Hence in our simulation we evaluate how of-
ten the endpoint ADs have to use a TD* path to reach
each other. The evaluation proceeds as follows: we ran-
domly select 1000 pairs of source and destination ADs.
For every pair, we compute the BGP path between the
pair as well as the SCI-FI up-paths of the source and the
destination ADs when k → ∞, i.e., the full up-paths in
their respective local trust domain (not the TD* univer-
sal domain). Again, considering full up-paths eliminates
the uncertainty and result variance due to the k-path se-
lection policies. Then we check whether the BGP path
between the pair is contained, i.e., in the union of the
up-paths of the source and the destination. The ratio of
contained paths represents the expressiveness without us-
ing the TD* domain. We also evaluate the expressiveness
as a function of k based on our simple k-path selection al-
gorithm to demonstrate the practicability of SCI-FI and
the trend as k increases. Fig. 7 summarizes the results
of SCI-FI’s expressiveness experiments, from which we
can see that with only a small k = 5 SCI-FI can already
capture more than 85% of BGP path diversity without
rescoping to the TD* domain.
10.4 Security
In Section 9, we have discussed attacks that are natu-
rally infeasible in our architecture. In this section, we
quantitatively investigate how severe such attacks are in
networks lacking our well-defined properties. As an ex-
ample, we consider the impact of long-standing wormhole
attacks when TD isolation is not enforced, i.e., an AD in
13
 0.8 AfriNIC
ratio of unavoidable hijacked traffic
ARIN
0.7 APNIC
LACNIC
0.6 RIPE NCC
0.5
0.4
0.3
0.2
0.1
0 0
2 4 6 8 10 12 14 16
number of malicoius ASes
Figure 8: Impact of wormhole attacks for S-BGP, which are intrinsically mitigated in SCI-FI.
1 with TD*
w/o TD*
0.95
expressiveness
0.9
0.85
0.8
0.75
0.7
1 2 3 4
Figure 7: Measurement results of SCI-FI expressiveness.
TD1 would prefer a route through an untrusted TD2 as
long as the route is shorter than others that are contained
entirely in its trust domain. In a wormhole attack, the
attacker attempts to attract routes by announcing a non-
existing shortcut (or “wormhole”). Clearly, in SCI-FI, it
is infeasible for an outsider to pull traffic out of a TD
with a strong isolation property, whereas in TDs where
no isolation is enforced, a wormhole residing in any corner
of the Internet can possibly attract a significant portion
of traffic from these TDs.
In our simulation, a wormhole attack is done by a group
of colluding ADs that announce a minimum cost to route
between each other. We consider such attacks by the most
influential ADs in a compromised domain, except the TD
Core ADs that we assume trusted. The influence score of
an AD in T Dx is evaluated by the number of ADs that see
that AD as the shortest path to T Dx . The observation is
that the attacker would be able to compromise a handful
of the ADs in a less trustworthy domain. We apply two
metrics to evaluate the power of route attraction attacks
by a small number of compromised ADs. These two met-
rics are ratio of unavoidable paths and ratio of attractable
paths. Unavoidable means that the normal BGP traffic
traverses a passive attacker that does not announce bogus
short paths. Attractable means that an active attacker
that sends bogus route announcements can attract BGP
routes that would otherwise go through some benign ADs.
Given a set of compromised ADs, our simulation re-
peats 1000 random selections of source and destination
0.14 0.8
0.12 0.7
ratio of total hijacked traffic
ratio of attractable traffic
0.6
0.1
0.5
0.08
0.4
0.06
0.3
0.04
0.2
0.02 0.1
0
18 20 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20
number of malicoius ASes number of malicoius ASes
ADs. For each simulation we measure if any of the com-
promised nodes would see the packet, in which case the
route is passively compromised (unavoidable). For the
active case (attractable), we simply allow the compro-
mised nodes to form a wormhole between each other by
announcing bogus paths and only count ”attracted” paths
that result from these bogus path announcements.
Fig. 8 shows the impact of malicious ADs on the path-
vector-based routing protocols without isolation. The x-
axis represents the number of ADs that collude and at-
5 6 7 8 9 10 all tempt to attract traffic. A value of i on the x-axis means
K that we select the i most influential ADs, where the in-
fluence score of an AD is evaluated based on how many
other AD nodes would consider this AD the shortest hop
to reach outside their respective TD. Each of the data
points is an average over 1000 runs of simulation with
randomly selected pairs of source and destination. The
result shows that without strong isolation the Internet is
extremely fragile because the attacker can control over
50% of Internet traffic with as few as five compromised
ADs. In contrast, by designing around the isolation, con-
trol, and scalability principles, SCI-FI can intrinsically
prevent these attacks as analyzed in Section 9.
11 Related Work
While no previously proposed solution seamlessly ad-
dresses all the security issues SCI-FI addresses simulta-
neously, prior work falls into a few distinct categories.
Routing security. Goldberg et al. analyze the weak-
nesses of BGP and S-BGP, quantifying their efficacy in
defending against traffic attraction attacks. Indeed, ex-
isting secure routing protocols such as soBGP [24], ps-
BGP [25], and SPV [26] only address the security of route
announcement semantics, which, at best, only guarantees
the paths are topologically valid but fails to to ensure the
logical trustworthiness and contractual legitimacy of the
routes.
Routing flexibility and scalability. Researchers have
proposed many inter-domain routing protocols attempt-
ing to improve the security of the current routing stan-
dard from a system’s point of view [27, 28, 29]. Also,
while there abound next-generation interdomain routing
proposals [3, 4, 5, 6, 7, 8, 9, 10] none of these proposals
14
focus on providing inherent routing security. In contrast,
we propose a new architecture that solves the attraction
weaknesses in S-BGP, intrinsically eliminates IP spoofing,
and easily extends to provide DDOS limiting.
DoS resilience. Many capability-based architectures
and forwarding protocols such as SIFF [20], TVA [30],
and NetFence [31] employ access control and rate control
to regulate inbound traffic to the victim destination. In
addition, other DoS-resilience architectures rely on build-
ing overlay networks (such as Phalanx [32]) to empower
the destination with inbound traffic control. As a sepa-
rate line of research, AIP [23] provides a building block –
source accountability – for defending against DOS and IP
spoofing by the use of self-certifying addresses. However,
all of these proposals simply provide a patch to the exist-
ing Internet. Instead we argue the necessity of an entirely
new architecture to achieve strong security properties.
12 Conclusion
We have presented SCI-FI, an architecture for supporting
well-principled isolation in Internet routing. The archi-
tecture provides a number of useful properties to inter-
domain routing including path choice, local accountabil-
ity, active route control at both endpoints, flexible path
choice without exposing ISPs to source-routing attacks,
and effective isolation which allows endpoints to estab-
lish routes limited to using a certain subset of ADs on
the Internet, which involves computations that only in-
volve those ADs in such a way that ADs outside of the
set are unable to influence the result. Rather than rely
on add-on mechanisms for security, each of these proper-
ties is provided as a natural outcome of structuring and
controlling the flow of the protocol computation in such
a way that it respects and reinforces real-world relation-
ships and dependencies.
Our evaluation results show that on current Internet
structure, SCI-FI incurs only about one additional hop in
average routing path length. With a small number of up-
paths (e.g., k=5), SCI-FI can capture more than 85% of
current BGP paths, thus exhibiting high path expressive-
ness with small overhead. Thanks to the strong security
properties as an intrinsic consequence of built-in isola-
tion, scalability, and control, our evaluation results show
that SCI-FI can improve end-to-end availability by up to
80% compared to existing (secure) interdomain routing
protocols.
The SCI-FI architecture demonstrates the importance
of building in good control and isolation properties in
a network protocol for a large number of heterogenous
principals. Control properties empower the principals
that have the highest stake in the communication; while
isolation protects the network elements that facilitate
this communication establishment from external influ-
ence. Together, these elements allow endpoints and net-
work providers to take an active role in explicitly deter-
15
mining the trusted elements in each route computation.
This makes it feasible to reason about the assurance level
of a given path, and to generate different routes with re-
silience and security appropriate to each application.
References
[1] “Insecure routing redirects youtube to pakistan,” Febru-
ary 2008, http://arstechnica.com/old/content/2008/02/
insecure-routing-redirects-youtube-to-pakistan.ars.
[2] S. Goldberg, M. Schapira, P. Hummon, and J. Rexford, “How
secure are secure interdomain routing protocols,” SIGCOMM
Comput. Commun. Rev., vol. 40, no. 4, pp. 87–98, 2010.
[3] P. B. Godfrey, I. Ganichev, S. Shenker, and I. Stoica, “Pathlet
routing,” in In Proc. SIGCOMM Workshop on Hot Topics in
Networking, 2008.
[4] M. Motiwala, M. Elmore, N. Feamster, and S. Vempala, “Path
splicing,” in ACM SIGCOMM, 2008.
[5] W. Xu and J. Rexford, “MIRO: Multi-path Interdomain Rout-
ing,” in ACM SIGCOMM, 2006.
[6] N. Kushman, S. Kandula, D. Katabi, and B. M. Maggs, “R-
BGP: Staying Connected In a Connected World,” in USENIX
NSDI, 2007.
[7] X. Zhang, A. Perrig, and H. Zhang, “Centaur: A hybrid ap-
proach for reliable policy-based routing,” in Proceedings of the
International Conference on Distributed Computing Systems
(ICDCS), Jun. 2009.
[8] X. Yang and D. Wetherall, “Source Selectable Path Diversity
via Routing Deflections,” in ACM SIGCOMM, 2006.
[9] D. Wendlandt, I. Avramopoulos, D. Andersen, and J. Rexford,
“Don’t secure routing protocols, secure data delivery,” in ACM
Hotnets, 2006.
[10] X. Zhang and A. Perrig, “Correlation-resilient path selection
in multi-path routing,” in IEEE Globecom, 2010.
[11] S. Kent, C. Lynn, J. Mikkelson, and K. Seo, “Secure border
gateway protocol (S-BGP) — real world performance and de-
ployment issues,” in Symposium on Network and Distributed
Systems Security (NDSS), Feb. 2000.
[12] M. Caesar and J. Rexford, “BGP routing policies in ISP net-
works,” IEEE Network Magazine, vol. Special issue on inter-
domain Routing, Dec 2005.
[13] L. Gao and J. Rexford, “Stable internet routing without global
coordination,” IEEE/ACM Trans. Netw., vol. 9, no. 6, pp.
681–692, 2001.
[14] A. Greenberg, G. Hjalmtysson, D. A. Maltz, A. Myers, J. Rex-
ford, G. Xie, H. Yan, J. Zhan, and H. Zhang, “A clean slate
4D approach to network control and management,” in ACM
SIGCOMM CCR, 2005.
[15] R. Mahajan, D. Wetherall, and T. E. Anderson, “Understand-
ing BGP misconfiguration,” in ACM SIGCOMM, 2002.
[16] X. Yang, “NIRA: A new routing architecture,” in ACM SIG-
COMM FDNA Workshop, 2003.
[17] M. Gerla, X. Hong, and G. Pei, “Landmark routing for large
ad hoc wireless networks,” 2000.
[18] P. Smith, “Internet routing table analysis update,” www.
sanog.org/resources/sanog14/sanog14-pfs-RoutingTable.pdf,
2009.
[19] R. Mahajan, D. Wetherall, and T. Anderson, “Understanding
bgp misconfiguration,” in ACM SIGCOMM, 2002.
[20] A. Yaar, A. Perrig, and D. Song, “SIFF: A stateless Internet
flow filter to mitigate DDoS flooding attacks,” in IEEE Sym-
posium on Security and Privacy, 2004.
[21] X. Liu, X. Yang, and Y. Lu, “To filter or to authorize:
Network-layer dos defense against multimillion-node botnets,”
in Proceedings of ACM SIGCOMM, 2008.
[22] B. Parno, D. Wendlandt, E. Shi, A. Perrig, B. Maggs, and Y.-
C. Hu, “Portcullis: Protecting connection setup from denial-of-
capability attacks,” in Proceedings of ACM SIGCOMM, 2007.
[23] D. G. Andersen, H. Balakrishnan, N. Feamster, T. Koponen,
 D. Moon, and S. Shenker, “Accountable Internet Protocol
(AIP),” in Proc. ACM SIGCOMM, Aug. 2008.
[24] R. White, “Securing BGP through secure origin BGP,” Cisco
Internet Protocol Journal, Tech. Rep., Sep. 2003.
[25] T. Wan, E. Kranakis, and P. van Oorschot, “Pretty secure
BGP (psBGP),” in Proceedings of Symposium on Network and
Distributed System Security (NDSS’05), 2005.
[26] Y.-C. Hu, A. Perrig, and M. Sirbu, “SPV: Secure path vec-
tor routing for securing BGP,” in Proceedings of ACM SIG-
COMM, Sep. 2004.
[27] K. Butler, P. McDaniel, and W. Aiello, “Optimizing bgp se-
curity by exploiting path stability,” in Proceedings of the 13th
ACM conference on Computer and communications security,
ser. CCS ’06, 2006, pp. 298–310.
[28] J. Karlin, S. Forrest, and J. Rexford, “Pretty good bgp: Im-
proving bgp by cautiously adopting routes,” in Proceedings of
the Proceedings of the 2006 IEEE International Conference on
Network Protocols, Washington, DC, USA, 2006, pp. 290–299.
[29] J. P. John, E. Katz-Bassett, A. Krishnamurthy, T. Anderson,
and A. Venkataramani, “Consensus routing: the internet as a
distributed system,” in Proceedings of the 5th USENIX Sym-
posium on Networked Systems Design and Implementation,
ser. NSDI’08, 2008, pp. 351–364.
[30] X. Yang, D. Wetherall, and T. Anderson, “TVA: A DoS-
limiting network architecture,” in IEEE/ACM Transactions
on Networking, 2008.
[31] X. Liu, X. Yang, and Y. Xia, “NetFence: Preventing internet
denial of service from inside out,” in ACM SIGCOMM, 2010.
[32] C. Dixon, T. Anderson, and A. Krishnamurthy, “Phalanx:
Withstanding multimillion-node botnets,” in USENIX NSDI,
2008.

16