ISP Core Network

Architecture
Anand Kumar
AGM (Broadband)
Broadband Faculty, ALTTC
 ISP Network – Major Components

 Core routers:
 These are the devices with higher performance and
functionality.
 They have high speed trunk connections.
 Access routers:
 These routers are with high port density and mainly
 forms the access /edge and distribution/aggregation
 Network
 Gateway routers :
 The main functionality of this devices is to have
 connectivity to other Providers.
 Service routers :
 These routers are mainly categorised for hosting and
servers
 MPLS
Multiprotocol Label Switching
 Agenda

 Drawbacks of Traditional IP Routing

 Basic MPLS Concepts
 Traffic Engineering with MPLS
 MPLS Architecture
 MPLS Labels
 Label Switch Routers
 Virtual Private Network
 Drawbacks of Traditional IP
Forwarding
Routing protocols are used to distribute Layer
3 routing information.
Forwarding is based on the destination
address only.
Routing lookups are performed on every hop.
 IP Packet Forwarding Example
134.5.6.1

Routing Table
Destination Next Hop

134.5/16 134.5.6.1 134.5.1.5

200.3.2/24 50.50.50.2

Routing lookup

200.3.2.7 2 Routing lookup

50.50.50.2
200.3.2.7
12.29.31.1 Routing lookup 30.30.30.2
20.20.20.2
Routing Table 3 5
Destination Next Hop 200.3.2.7 200.3.2.7
10.10.10.2 Routing lookup
134.5/16 12.29.31.5

200.3.2/24 10.10.10.2

Routing Table Routing Table 200.3.2.1 200.3.2.7

Destination Next Hop Destination Next Hop

134.5/16 12.29.31.9 134.5/16 12.29.31.4

200.3.2/24 20.20.20.2 200.3.2/24 30.30.30.2

 Drawbacks of Traditional IP Forwarding…
Traffic Engineering

– Most traffic goes between large sites A and B, and uses only the primary
link.
 Basic MPLS Concepts
MPLS is a new forwarding mechanism in
which packets are forwarded based on labels.
Labels usually correspond to IP destination
networks
MPLS was designed to support forwarding of
other protocols as well.
 What is MPLS ?

 MPLS is a type of data-carrying technique for high-

performance telecommunications network
 Multi protocol label switching (MPLS) is a versatile
solution to address the problems faced by present
day networks – speed, scalability,quality of service
(QoS) management, and traffic engineering.
 MPLS is a scalable, protocol-independent transport.
 MPLS can encapsulate packets of various network
protocols, hence its name "multiprotocol
 MPLS supports a range of access technologies,
including T1/E1 , Frame Relay etc.
 MPLS Model
Position of MPLS in TCP/IP Model
Layer 4-5 (Transport, Application)
Layer 3 (Network)
Layer 2.5 (?) (MPLS)
Layer 2 (Data link )
Layer 1 (Physical)
 MPLS Labels (Cont.)
MPLS uses a 32-bit label field that is inserted between
Layer 2 and Layer 3 headers
 MPLS Labels (Cont.)
Label Format

• MPLS uses a 32-bit label field that contains the following

information:
– 20-bit label
– 3-bit experimental field
– 1-bit bottom-of-stack indicator
– 8-bit TTL field
 MPLS Label Stack
• Bottom-of-stack bit indicates whether the next header is another
label or a Layer 3 header.
• Receiving router uses the top label only.
• Top label or inner label is called VPN Label
• Bottom label is called LSP label.
 MPLS Label Stack (Cont.)
• Usually only one label is assigned to a packet.
• The following scenarios may produce more than one
label:
– MPLS VPNs (two labels): The top label points to the egress
router and the second/inner label identifies the VPN.
– MPLS TE (two or more labels): The top label points to the
endpoint of the traffic engineering tunnel and the second
label points to the destination.
 MPLS Technology

EDGE CORE Gate Way

Provider Edge Router Provider Router BSNL is having 11
Connect to: Core Connect to: Core Gateway Chain
routers on one side routers on one side routers located in 7
and Customer IP and PE routers on places. One side Core
Networks( on another side. Each PE and other side IBP,
Customer Edger is minimum Dual Peering links
routers) Parented including NIXI.

Best in class technology routers are Deployed in BSNL and

AMC&OEM support is ensured at the procurement stage
itself. As a continuous process Tech refresh is done on all the
MPLS devices to keep them up to date and ensures latest
services are available to Customers.

15
 How LDP Works
Net: Net: Net:
10.10.10.0 10.10.10.0 10.10.10.0
Label: 45 Label: 35 Label: 25
Label
20.20.20.0/24 P1 P2 PE-2
Request to
PE-1 reach
10.10.10.0

1 2 3 4 5 6 7 2
8
LFIB Table LFIB Table LFIB Table LFIB Table
Local Out/NH 10.10.10.0/24
Local Out/NH Local Out/NH local Out/NH
55 45/3 25
45 35/5 35 25/7

IP Label Label IP
Routing Switching Switching Routing
ROUTE AT EDGE,
SWITCH IN CORE
 How LDP Works
Net: Net: Net:
10.10.10.0 10.10.10.0 10.10.10.0
Label: 45 Label: 35 Label: 25
Label
20.20.20.0/24 P1 P2 PE-2
Request to
PE-1 reach
10.10.10.0

1 2 3 4 5 6 7 2
8
LFIB Table LFIB Table LFIB Table LIB Table
Local Out 10.10.10.0/24
Local Out Local Out local Out
55 45 25
45 35 35 25/PR-2

IP Label Label IP
Routing Switching Switching Routing
ROUTE AT EDGE,
SWITCH IN CORE
 How MPLS Works

remove label
Add label at at the egress
the ingress PE SWAP SWAP PE
PUSH POP
20.20.20.0 P P PE
10.10.10.0
PE

1 2 3 4 35 6 7 8

IP IP 45 IP 35 IP 25 IP

IP Label Label IP
Routing Switching Switching Routing
ROUTE AT EDGE,
SWITCH IN CORE
 PHP : Penultimate Hop POP
Net: Net: Net:
10.10.10.0 10.10.10.0 10.10.10.0
Label: 45 Label: 35 Label: PoP or 3
Label
20.20.20.0/24 P P PE
Request to
PE reach
10.10.10.0

1 2 3 4 5 6 7 2
8
LFIB Table LFIB Table LFIB Table LFIB Table
Local Out 10.10.10.0/24
Local Out Local Out local Out
55 45 PoP
45 35 35 PoP

IP Label Label IP
Routing Switching Switching Routing
ROUTE AT EDGE,
SWITCH IN CORE
 PHP : Penultimate Hop POP

Add label at remove label

the ingress PE SWAP POP
PUSH
P P PE
PE

IP IP 45 IP 35 IP IP

IP Label Label IP
Routing Switching Switching Routing
ROUTE AT EDGE,
SWITCH IN CORE
 MPLS v IP Routing
Source Destination
IP Routing Domain

Examine IP header Examine IP header Examine IP header Examine IP header

Ingress Egress
Source Destination
LSR MPLS Domain LSR

Examine IP header Examine IP header

Assign to FEC Label swap Label POP Assign to FEC
Forward Forward Forward IP Forward
packet
 ROUTING INFO

Why MPLS customer prefers?

ROUTING
Traditional IP Network Routing
CIP CIP

IP Multiple network checking

(a match with longest network Mask wins)

SWITCHING

SWITCHING INFO

New Reference is
Applied to packet

ATM ATM ATM ATM ATM ATM

22
 Label Switch Path (LSP)

IGP domain with a label

distribution protocol

LSP follows IGP shortest path

• LSPs are derived from IGP routing information

• LSPs are unidirectional
Return traffic takes another LSP
 Traffic Engineering with MPLS

• Traffic can be forwarded based on other parameters (QoS,

source, ...).
• Load sharing across unequal paths can be achieved.
 Haldwani(1+1) HISAR kARNAL Rohtak FARIDABAD(1+
1)
AMBALA(2+2)
Srinagar(garhwal)
Nainital GURGAUN(1+2)
Haridwar
Dharamsa
Dehradun(1+
1)
Almorah NTR la Hamirpur

Shimla(2+1)
Moradabad
Jammu(1+1
Noida(7+2) )

Muzaffarnagar(1+1) Srinaga
CHANDIGARH r
Chandigarh(5+2)
Meerut(1+1
) DEHRDUN
Bathind
JAIPUR
Ghaziabad(1+1) a
JALANDHAR
Hoshiarpur
Bareilly

Agra(1+2) Jalandhar(2+
2)
JAMMU
NEWDELHI(3+2) NOIDA
NEWDELHI Ludhiana(1+1
)
Varanasi(1+1 Pathankoat
) Agra

Mirzapur Ropar
LUCKNOW(1+1)

Lucknow(3+ Ajmer(1+
2) AHMEDABAD 1)
Kanpur(1+1
ALLAHABAD
) Alwar(1+1
PATNA
Sitapur )

JODHPUR Bhilwar
Gorakhpur a
INDORE
Faziabad Bikaner

Jaipur(2+2)
Jhansi(1+1) Jo
Jodhpur(1+
1)
Basti
Kota(1+1)
Azamgarh Udaipur(1+ Nagaur
Allahabad(1+1 1)
) Sangru
r Patial Ferozpur Amritsa Sriganganagar
 MPLS Strength
All India MPLS NOC is situated at Bangalore and DR NOC is located
at Pune. Proactive Monitoring and Corrective Actions including
provisioning of Circuits are done from Bangalore NOC. Our Strength:-

51 Core routers are deployed at 44 Locations, 11

IGW+ 11 IGWPE Gate Way Routers deployed at
CORE ROUTERS
7 Locations.
.

450 EDGE routers are deployed at 204 locations

EDGE ROUTERS across India. Most recent MPLS PoP addition
being Port Blair, A & N.

1,90,000 Links are provisioned and working in

Back Bone Links BSNL.

26
 MPLS Strength
1 Tbps Internet Bandwidth is available to
BSNL customers.
Internet Bandwidth [Google 260G, Akamai 170G,
International bandwidth 400G, Others
170G

[Customer Links = 1,75,000 & Service

Links = 15,000].
BSNL has tie up with M/s Vodafone for
Customer Links international VPNs. Ex Canara Bank is
availing now.
ILL and VPNoBB links can be provided
from BSNL with dedicated band width

27
 Customer Connectivity

BSNL Edge
Router
MPLS Cloud

L3 Switch
MPL
S

WIMA
IP CMTS X
BNG TAX
NTU / CPE / System

Customer NTU / CPE / System BSNL Services

 Gateway Connectivity
Chandigarh Chandigarh
Edge Router Core Router
2 * 10G Inter National
Bandwidth
Colocated Core-Edge Providers

8 * 10G New Delhi

GATEWAY

New Delhi
Core Router NIXI, Google,
Remote Edge Router Yahoo, Facebook
Gurgaon Flipcart,
Edge Router Railtel,Microsoft,
IRCTC
 MPLS Services Offered to Customers
Quality of Service: The QoS feature of MPLS networks
ensures a measure of guarantee in packet delivery. In case of
choked links, packets are dropped/lost. In networks without QoS,
the packets will not have any priority and all packets are equally
liable to be dropped. With QoS implemented, different priorities
are assigned so that packets with lower priority are dropped first.
The order of priority in BSNL MPLS network is as below:-

• MANAGE - Reserved for Protocol traffic

• PLATINUM - Reserved for Voice
• GOLD - Highest priority - Open for commercial MPLS VPN
• SILVER - Medium priority - Open for commercial MPLS VPN
• BRONZE - Low priority - Open for commercial MPLS VPN
• BEST-EFFORT - Lowest priority - for Internet traffic
 What is Quality of Service
Desktop
Conferencing,
Distance Learning

Mission-Critical
Applications

E-Mail

FTP
 MPLS NOC
Big Wall Screen to Monitor Devices & Links

Central logging server

IGWPE BMAP
Router NMS (HPOV NNM)

H P E SA

Core Router

Fire Wall

Edge Router
SLA Monitoring
(HP-OV)
LAN Switch

Gigabit Ethernet Fast Ethernet

24X7 Monitoring
32
 Benefits of MPLS
 Ultra fast forwarding
 IP Traffic Engineering
 Virtual Private Networks
 QOS
 Reliable
 Secured
 Cost Effective
 Scalable
 Manageable
 Easy Maintenance
 BSNL MPLS Special

 MPLS NOC is having Test Bed MPLS Network set

up which is widely used for testing the new
services and new technologies before putting on
to commercial use. Customers set up is made
and demonstrated to over come technical
challenges.
 IXIA Traffic generators/Analyzers are available to
demonstrate Band width, end to end to the
customers.
 NMS with portal to monitor and manage customer
links from their terminal itself.
 SLA monitoring available for customers.

34
 New developments in MPLS Core in BSNL

 BSNL is going in for very high capacity Super Core routers

 These routers will have 100G interfaces besides 10G
 The 100G transport network is ready in the form of OTN
 The Super Core will be have SDN capabilities
 SDN Controller from M/s NOKIA will interact with all devices
on the network (installation phase)
 New devices will support Segment Routing using which the
SDN Controller will be able to allow for feature rich services
for customers such as congestion free routing
MPLS-VPN Security
 Private Networks

Organization A
Site 4

Organization B
Site 1

Organization A
Site 3

Organization A
Site 1

Organization B
Site 3

Organization B
Site 2 Leased Lines
Organization A
Site 2
 Private Network
• Advantages:
– Leased lines are secured
– Privacy and QoS Guarnteed
• Disadvantages
– Leased lines are very expensive
– No of links required grows exponentially if full mesh
connectivity is required and network expands.
– More nos of CPE ports are required
– Network complexity increases as network grows. All
existing sites requires reconfiguration in case of a new site
addition.
 Complexity of Customers Network with Point to Point
Leased Lines
Ahmedabad Lucknow

Mumbai

Pune Delhi

Leased lines from BSNL

Bangalore Kolkotta

Ernakulam Chennai
Hyderabad

Links
 How it looks – in the MPLS environment

Ahmedabad

Lucknow

Mumbai

Delhi
Pune

BSNLs MPLS VPN Network

Bangalore
Kolkotta

Ernakulam Chennai

Hyderabad
Router
Links
 Virtual Private Network Defined

Customer Connectivity Deployed on a Shared

Infrastructure with the Same Policies as a Private
Network

Intranet Site-to-Site VPN

SP Shared Branch offices
Network Intranet Ptoint to Multi Point
VPN Head office & branch offices
Extranet VPN
Business-to-business
Inter As VPN VPN
Customers on Different ISP
 MPLS -VPN
(Layer-3 & Layer-2)
 Layer -3 VPN 10.30.0.1 /16

Chennai-PE
Ser 4/0/0:0
Lo0 7.254
MP-iBGP 16.62 16.65 16.66
Static Default
IBM-CE-3

Mumbai-P 16.61
MP-iBGP
POS 6/0 Delhi-P
16.42 16.41
10.10.0.1 /16 16.49 20.50.0.1 /16
Pune-PE
Kolkata-PE
Lo0 3.254 16.50 OSPF- Area 0 16.46
FE 0/0 FE1/1 FE1 FE
Lo0 3.254 Lo0 1.254
16..70 16.69 /1 0/0
16.17 16.18
16.5 16.14
MP-iBGP
IBM-CE-1 Default
Static Static
Default Sun-CE-2

16.6 16.13

STM 1
Sun-CE-1 IBM-CE-2
Gigabit Ethernet
20.40.0.1 /16 10.20.0.1 /16 MPLS Domain
E-1 (2Mbps)
 Layer 3 Intranet MPLS VPN service

Customer C
Site 3

VRF for
customer A
PE Router Customer A
C CC Site 2
CC C

AA
A A
AA A
Customer A A A
Site 1 A C CC
CC C
Customer C
C CC Site 2
CC C

Customer C
Site 1
 Layer 3 Intranet MPLS VPN service
Mesh Connectivity
Customer A
Site 3 Customer A
Customer A Site 2
Site 4

C CC
CC C
C CC
C CC CC C
CC C
Customer A
Site 2
C CC
C CC CC C
CC C
Customer A C CC
Site 5 CC C
Customer Intranet VRF

Customer A
Site 1
 Layer 3 Intranet MPLS VPN service
Hub & Spoke
connectivity Customer A
Site 3 Customer A
Customer A Site 5
Site 4

C CC
CC C
C CC
C CC CC C
CC C

C CC
C CC CC C
CC C
Customer A C CC
Site 5 CC C Customer A
Site 2
Hub Site

Customer A
Site 1
 Layer 3 Extranet MPLS VPN Service

Customer 2
Site B
C1a
VRF C1a
C2b C1a
C2a C2b
C2b C2a

C2b VRF
C2a C2b C1b
C2b C2a C1a C1b
Customer 2 C1b C1a
Site A C2b Customer 1
C2b C1b C2b
Site B
C1a C1b
C1b C1a
VRF
Customer 1
Site A
Import/Export of routes
 Inter –AS-VPN
MP-eBGP

Mumbai P Delhi P
16.42 Gige 7/0/0 16.41

16.49 MPLS Label

16.46
16.50
RR RR
Gige 2/1 Gige 2/1
16..22 16.21 16.17 16.18
16.5 16.14

Ahmedabad P Pune P Kolkata P Guwahati P

AS-1000 MTNL AS-9829 BSNL
16.6 16.13

MP-iBGP MP-iBGP
STM 1 16.29 16.25
Bangalore PE Hyderabad PE
Gigabit Ethernet
eBGP
eBGP

E-1 (2Mbps)

Sun CE-1 Sun CE-2

MPLS Layer-2VPN
 Pseudo Wire Reference Model
Customer Customer
Site Site
PSN* Tunnel

Pseudo Wires AC
AC

PE
AC
PE AC
Customer Customer
Site Site
Emulated Service
 The architecture is based on the pseudowires.
 Pseudowire is a connection between PE to PE & emulates a wire that is carrying Layer -2
frames.
 The pseudowires carry the customer layer-2 traffic from PE to PE across the packet
switched network.
 Attachment Circuit , is a physical or logical circuit between PE and CE devices and used
in MPLS Layer 2 VPN

*PSN – Packet Switched Network

MPLS Security in BSNL
 Agenda
• Security features of MPLS technology
• Securing operations of MPLS NOC
 Security Features of MPLS Technology
Address Space Separation
64 bits 32 bits

Route Distinguisher IPv4 Address

VPN IPv4 Address

Within the MPLS core all addresses are unique due to the
Route Distinguisher
the address space between different VPNs is entirely
independent.
Different VPNs must be able to use the 10/8 network without
any interference

5
3
Security Features of MPLS Technology
Routing & Forwarding Separation
64 bits 32 bits

Route Distinguisher IPv4 Address

VPN IPv4 Address

Routing Seperation:
Every PE router maintains a separate Virtual Routing and
Forwarding instance (VRF) for each connected VPN & each VRF
has a RD (route distinguisher)
Because every VPN results in a separate VRF, there will be no
interferences between the VPNs on the PE router

5
4
 Security Features of MPLS Technology
Routing & Forwarding Separation
64 bits 32 bits

Route Distinguisher IPv4 Address

VPN IPv4 Address

Across the MPLS core to the other PE routers, this separation

is maintained by adding unique VPN identifiers in multiprotocol
BGP (MP BGP).
Thus, routing across an MPLS network is separate per VPN.
Forwarding Seperation: By the use of labels that are unique to
the VPNs configured.

5
5
 Security features of MPLS technology
• Option to hide the MPLS Core – by disabling TTL
propagation
 Security Features of MPLS Technology

Label Spoofing
 Label spoofing is not possible as it is a random
number pre-negotiated by devices through standard
protocols
 PE router expects IP packet from CE
 Labelled packets will be dropped
 Thus no spoofing possible
 Securing Operations of MPLS NOC

Device access
1. Only a predefined set of source IPs can
access the router
2. Device can’t be accessed from CE router
3. Authorised users with privilege levels
restrict operation to necessary minimum
4. All configuration is centralised
 Securing Operations of MPLS NOC

Service Provisioning
1. All provisioning of services is carried out through
an automated tool. (No CLI is used for customer
profile configuration).
2. The NMS tools of NOC are behind a firewall for
protection.
 Security Features of MPLS technology
Security of MPLS Core
 Specific Security Checks for Hardening
of MPLS Boxes
• Use of MD-5 authentication password for IGP and
LDP so that no rogue device can latch-on to the
network.
• Implemented CoPP to protect the CPU of the router
from getting overloaded due to malicious control
protocol traffic.
• TACACS deployed for all MPLS network elements –
login and privilege management is centralised
 Other security Measures in Place

• Regular change of device passwords

• Upgrade of the router software (IOS) as per the
security advisory of OEMs
• Application of security patches to our NMS servers
as per security advisories issued by agencies such as
CERT-IN
• Implementation of Security Policy
• Carrying out regular Security Audits
 What MPLS does not Provide
 Protection against Misconfigurations
 Deliberate or inadvertent misconfigurations from SP
staff may result in undesired behavior, including
severe security leaks
 To avoid the risk of misconfigurations, it is
important that the equipment is easy to configure,
and that SP staff has the appropriate training and
experience when configuring the network.
Thank you….