Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Authored by: Rick Mur - CCIE3 #21946 (R&S / SP / Storage), JNCIE-SP #851
CCIE Data Center Lab Preparation Detailed Solution Guide
IPexpert’s
Lab Preparation Detailed Solution Guide for
Cisco’s CCIE Data Center Lab
Before We Begin
This product is part of the IPexpert suite of materials that provide CCIE candidates and network
engineers with a comprehensive training program. For information about the full solution, contact an
IPexpert Training Advisor today.
Telephone: +1.810.326.1444
Email: sales@ipexpert.com
Congratulations! You now possess one of the ULTIMATE CCIETM Lab preparation and network
operation resources available today! This resource was produced by senior engineers, technical
instructors, and author boasting decades of internetworling experience. Although there is no way to
100% guarantee success rate on the CCIE™ Data Center Lab exam, we feel VERY confident that your
chances of passing the Lab will improve dramatically after completing this industry-recognized
Workbook!
IPexpert is proud to lead the industry with multiple support options at your disposal free of charge. Our
online communities have attracted a membership of over 20,000 of your peers from around the world!
At blog.ipexpert.com, you can keep up to date with everything IPexpert does and read the latest in
technical articles from world-renowned IPexpert instructors. At OnlineStudyList.com, you may subscribe
to multiple “SPAM-free,” moderated CCIE-focused email lists.
Feedback
Do you have a suggestion or other feedback regarding this book or other IPexpert products? At IPexpert,
we look to you – our valued clients – for the real world, frontline evaluation that we believe is necessary
so that we may always improve. Please send an email with your thoughts to feedback@ipexpert.com or
call 1.866.225.8064 (international callers dial +1.810.326.1444).
In addition, for those using this book as CCIETM preparation, when you pass the CCIETM Lab exam, we
want to hear about it! Email your CCIETM number to success@ipexpert.com and let us know how
IPexpert helped you succeed. We would like to send you a gift of thanks and congratulations.
IPexpert, Inc. is committed to developing the most effective Cisco CCIETM R&S, Security, Voice, Wireless
and Data Center Lab certification preparation tools available. Our team of certified networking
professionals develops the most up-to-date and comprehensive materials for networking certification,
including self-paced workbooks, online Cisco hardware rental, classroom training, online (distance
learning) instructor-led training, audio products, and video training materials. Unlike other certification-
training providers, we employ the most experienced and accomplished teams of experts to create,
maintain, and constantly update our products. At IPexpert, we are focus on making your CCIETM Lab
preparation more effective.
This book is carefully edited to ensure the accuracy of all content. Should you find any error whatsoever,
please email a page reference and detailed comment to wberrors@ipexpert.com. Your email will be
responded to promptly.
This is a legally binding agreement between you and IPEXPERT, the “Licensor,” from whom you have
licensed the IPEXPERT training materials (the “Training Materials”). By using the Training Materials, you
agree to be bound by the terms of this License, except to the extent these terms have been modified by
a written agreement (the “Governing Agreement”) signed by you (or the party that has licensed the
Training Materials for your use) and an executive officer of Licensor. If you do not agree to the License
terms, the Licensor is unwilling to license the Training Materials to you. In this event, you may not use
the Training Materials, and you should promptly contact the Licensor for return instructions.
The Training Materials shall be used by only ONE (1) INDIVIDUAL who shall be the sole individual
authorized to use the Training Materials throughout the term of this License.
The Training Materials are the property of IPEXPERT, Inc. ("IPEXPERT") and are protected by United
States and International copyright laws. All copyright, trademark, and other proprietary rights in the
Training Materials and in the Training Materials, text, graphics, design elements, audio, and all other
materials originated by IPEXPERT at its site, in its workbooks, scenarios and courses (the "IPEXPERT
Information") are reserved to IPEXPERT.
The Training Materials cannot be used by or transferred to any other person. You may not rent, lease,
loan, barter, sell or time-share the Training Materials or accompanying documentation. You may not
reverse engineer, decompile, or disassemble the Training Materials. You may not modify, or create
derivative works based upon the Training Materials in whole or in part. You may not reproduce, store,
upload, post, transmit, download or distribute in any form or by any means, electronic, mechanical,
recording or otherwise any part of the Training Materials and IPEXPERT Information other than printing
out or downloading portions of the text and images for your own personal, non-commercial use without
the prior written permission of IPEXPERT.
You shall observe copyright and other restrictions imposed by IPEXPERT. You may not use the Training
Materials or IPEXPERT Information in any manner that infringes the rights of any person or entity.
Exclusions of Warranties
THE TRAINING MATERIALS AND DOCUMENTATION ARE PROVIDED “AS IS.” LICENSOR HEREBY DISCLAIMS
ALL OTHER WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING WITHOUT LIMITATION, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. SOME STATES
DO NOT ALLOW THE LIMITATION OF INCIDENTAL DAMAGES OR LIMITATIONS ON HOW LONG AN
IMPLIED WARRANTY LASTS, SO THE ABOVE LIMITATIONS OR EXCLUSIONS MAY NOT APPLY TO YOU. This
agreement gives you specific legal rights, and you may have other rights that vary from state to state.
ANY ACTION ON ANY CLAIM AGAINST IPEXPERT MUST BE BROUGHT BY THE USER WITHIN ONE (1) YEAR
FOLLOWING THE DATE THE CLAIM FIRST ACCRUED, OR SHALL BE DEEMED WAIVED. IN NO EVENT WILL
THE LICENSOR’S LIABILITY UNDER, ARISING OUT OF, OR RELATING TO THIS AGREEMENT EXCEED THE
AMOUNT PAID TO LICENSOR FOR THE TRAINING MATERIALS. LICENSOR SHALL NOT BE LIABLE FOR ANY
SPECIAL, INCIDENTAL, INDIRECT, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, REGARDLESS OF WHETHER LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES. WITHOUT LIMITING THE FOREGOING, LICENSOR WILL NOT BE LIABLE FOR LOST
PROFITS, LOSS OF DATA, OR COSTS OF COVER.
Entire Agreement
This is the entire agreement between the parties and may not be modified except in writing signed by
both parties.
The Training Materials and accompanying documentation are “commercial computer Training
Materials” and “commercial computer Training Materials documentation,” respectively, pursuant to
DFAR Section 227.7202 and FAR Section 12.212, as applicable. Any use, modification, reproduction
release, performance, display, or disclosure of the Training Materials and accompanying documentation
by the U.S. Government shall be governed solely by the terms of this Agreement and shall be prohibited
except to the extent expressly permitted by the terms of this Agreement.
IF YOU DO NOT AGREE WITH THE ABOVE TERMS AND CONDITIONS, DO NOT OPEN OR USE THE
TRAINING MATERIALS AND CONTACT LICENSOR FOR INSTRUCTIONS ON RETURN OF THE TRAINING
MATERIAL
Contents
IPexpert’s ...................................................................................................................................................... 1
Lab Preparation Detailed Solution Guide for Cisco’s CCIE Data Center Lab............................................. 1
Before We Begin ....................................................................................................................................... 1
Feedback................................................................................................................................................... 2
Additional CCIETM Preparation Material ................................................................................................... 2
Issues with this Book ................................................................................................................................ 2
IPEXPERT END-USER LICENSE AGREEMENT.............................................................................................. 3
Copyright and Proprietary Rights ............................................................................................................. 3
Exclusions of Warranties .......................................................................................................................... 4
Choice of Law and Jurisdiction ................................................................................................................. 4
Limitation of Claims and Liability.............................................................................................................. 4
Entire Agreement ..................................................................................................................................... 5
U.S. Government - Restricted Rights ........................................................................................................ 5
Default Lab Topology ................................................................................................................................ 9
Default passwords and IP addresses ........................................................................................................ 9
Chapter 1: Introduction to CCIE Data Center.............................................................................................. 10
Who Should Read this Book? ................................................................................................................. 11
How to Use this Book ............................................................................................................................. 11
An Introduction to CCIE Data Center...................................................................................................... 11
Availability .............................................................................................................................................. 12
Written exam .......................................................................................................................................... 12
The current published reading list:......................................................................................................... 12
Lab exam................................................................................................................................................. 13
Software Versions................................................................................................................................... 13
CCIE Storage?.......................................................................................................................................... 13
What about P and A tracks? ................................................................................................................... 13
Troubleshooting ..................................................................................................................................... 14
An Introduction to the Proctor Labs CCIE Data Center hardware rack .................................................. 14
Software Versions................................................................................................................................... 17
Chapter 2: Data Center Networking Layer 2 Infrastructure ....................................................................... 18
(NX-OS) ........................................................................................................................................................ 18
Generic Lab topology.............................................................................................................................. 19
Introduction ............................................................................................................................................ 20
General Rules.......................................................................................................................................... 20
Solutions ................................................................................................................................................. 21
Task 1: General set-up ........................................................................................................................ 21
Task 2: Implement VLANs ................................................................................................................... 26
Task 3: Implement Private-VLANs....................................................................................................... 33
Task 4: Implement Rapid Spanning-Tree protocol ............................................................................. 41
Task 5: Implement Multiple Spanning-Tree protocol ......................................................................... 48
Task 6: Spanning-Tree and UDLD features ......................................................................................... 56
Task 7: Fabric Extenders ..................................................................................................................... 58
Task 8: Misc features .......................................................................................................................... 61
Chapter 3: Data Center Networking Layer 3 Infrastructure (NX-OS) .......................................................... 64
General Rules.......................................................................................................................................... 65
Solutions ................................................................................................................................................. 66
Chapter 1:
Introduction to CCIE
Data Center
Chapter 1: Introduction to CCIE Data Center introduces the team of authors, consultants, and editors
that completed this book and describes the book’s purpose. This chapter also provides suggestions for
the usage of this written work.
The final chapters conclude the book with sample lab scenarios that provide a full scale lab exam as you
will see it when you take the actual test. The Detailed Solutions Guide then provides a well-designed
approach for troubleshooting each major task and offers detailed explanations. The text provides
reference guides for the most popular and powerful show and debug commands for a specific
technology.
Each chapter uses specific initial configurations on the specific chapter. Readers may download initial
configurations, or install them in a simple Graphical User Interface (GUI) on www.proctorlabs.com.
Students are encouraged to follow along on a rack of equipment for every section of every chapter. This
really enhances and strengthens the learning process.
The scope of the exam is pretty much based on the usual suspects, so in summary you should be aware
of the:
Availability
The live exam is available from September 1st.
Written exam
The written exam has an extensive blueprint published to Cisco Learning Network (CLN) including a
reading list.
Please find the extensive blueprint published by Cisco on the bottom of this blog post.
Lab exam
There is not much information available regarding the lab exam. Availability is not mentioned. There is
however information regarding the hardware list and this is an immense list of expensive hardware you
require:
Software Versions
CCIE Storage?
There are currently no plans for replacing CCIE Storage for CCIE Datacenter. Because of this, there will
not be a large focus on MDS/FC configuration as there is another track for that.
Troubleshooting
Troubleshooting will be a big part of the exam, which is also pretty clear in the blueprint. There is no
confirmation yet how this will be introduced, either using tickets in the CCIE R&S or just by pre-
configuration on the lab. I can imagine that they pre-configured a broken Nexus 1000V on an ESX
installation on one of the JBODs. More information on how this troubleshooting is done will be available
during other Q&A sessions. The implication is that it might be trouble tickets like the CCIE R&S.
Our CCIE Data Center rack layout is based on the very limited information that has been made available
by Cisco. IPexpert has been in close contact with the people involved in creating this lab exam, and
therefore the layout of the rack is based on some early examples and the published components and
software version blueprint.
As you will see the topology is very much based on a common datacenter design and has more 'static'
layout than other CCIE tracks.
The Nexus 7000 will be configured with VDC's to simulate various different topologies and create
multiple 'core switch' layers within the network.
Nexus 5548 will be used as a 'distribution' layer within the datacenter network. The Nexus 2k's can be
configured as FEX for the Nexus 7000; Nexus 5000 and the Fabric Interconnects of the UCS system to
connect the UCS C-series rack mount servers. The VDC's are a major component in the network as the
number of devices is limited and the connectivity is very much based on a best practice design.
The below drawing illustrates an example topology from our new CCIE Data Center lab preparation
workbook which is currently under development.
All these interconnections and switches are based within a single physical chassis with complete
separation of the control and data plane protocols!
The MDS switches used in the lab are capable of a ton of features. The blueprint however only describes
certain fibre-channel features which are considered 'basic' features like zoning, VSANs, oversubscription
and ISLs. The other major topic on the blueprint is Fibre Channel Expansion over FCIP and iSCSI. These
features are the IP features supported by the MDS platform. The 1G Ethernet connections are connected
to the Nexus switches for testing the expansion features. Through that connection it's possible to
connect the MDS switches across another connection than Fibre Channel. As the CCIE Storage track is
Copyright © by IPexpert. All rights reserved. 15
CCIE Data Center Lab Preparation Detailed Solution Guide
not being replaced by the CCIE Data Center the focus on Storage Networking (SAN) features is not that
big. The major topics are more in the features that aren't tested in any other CCIE track.
The JBODs mentioned in this list represent just plain simple hard-disks that are connected via Fibre
Channel. They are used later as shared storage for the UCS system.
The third major component within the hardware blueprint is the Unified Computing System (UCS).
This is based on the C-series rackmount servers, connected to the Fabric Interconnects so the C-series
can also be managed from the central UCS manager the same as the Blade chassis is managed.
The blades are equipped with different NICs. This also means a little different configuration. The VIC
cards are the most interesting ones as they can virtualize NICs to present to the OS.
Ones inside the blades there is a pre-installed VMware ESX(i) environment with a Nexus 1000v
distributed virtual switch. As this is a Cisco lab exam, you are not required to know anything about
VMware. Of course you will need to be able to install this environment in your possible own lab, but
when you step into the lab you will face a pre-installed VMware and 1000V. After that, the switch is not
configured and you are required to configure it.
The final topic on the blueprint is called ANS (Application Networking Services). This means an ACE
appliance is in your lab that you will need to configure. There is not much very interesting going on there
and you will not see a lot of points on that appliance. You will need to know the topics as described on
the lab blueprint and our workbook will focus a whole section on these specific topics.
The last components are used for management. You will not be configuring these devices, but just using
them from your student workstation to access the network.
What is not mentioned on the hardware blueprint list is that you will also need to be able to configure
(or set-up) the DCNM software as is being given by Cisco when you purchase enough Nexus equipment.
Again this is not extremely difficult, but you need to be aware of the basic configuration items related to
this software.
Software Versions
Above you'll find a reference overview of the used software versions. The exact versions are still
unknown where we might be using newer software versions as our IPexpert lab will be using quite new
hardware for virtualization purposes. Within the Nexus 7000 we will be using the new Supervisor 2E,
meaning that we are able to build 8 VDC's and 1 management VDC meaning we have enough flexibility
for some challenging topologies!
The next chapter of this workbook, Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS)
begins with the initial topic on the CCIE Data Center Blueprint regarding layer 2 switching, VLANs,
Private-VLANs, Spanning-Tree and other layer 2 features on the NX-OS platform.
Chapter 2: Data
Center Networking
Layer 2
Infrastructure
(NX-OS)
Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS) is intended to let you be familiar
with the NX-OS CLI on the Nexus switches and afterwards configure Layer 2 Ethernet features on the
physical Nexus switches within the topology as shown at the beginning of this workbook. We highly
recommend to create your own diagram at the beginning of each lab so you are able to draw on your
own diagram, making it much easier when you step into the real lab. Our devices start with a blank
configuration, which will not be the case when you are in the real lab. Then devices are staged with
configuration containing usernames/passwords, management IP addressing, core IP addressing and
(possible) errors.
Introduction
This first lab is intended to let you be familiar with the NX-OS CLI on the Nexus switches and afterwards
configure Layer 2 Ethernet features on the physical Nexus switches within the topology as shown at the
beginning of this workbook.
We highly recommend to create your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab.
Our devices start with a blank configuration, which will not be the case when you are in the real lab.
Then devices are staged with configuration containing usernames/passwords, management IP
addressing, core IP addressing and (possible) errors.
General Rules
• Try to diagram out the task. Draw your own connections the way you like it
• Take a very close read of the tasks to ensure you don’t miss any points during grading!
• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Solutions
INIT: version 2
Checking all filesystems..r.r.r..r done.
Loading system software
Do you want to enforce secure password standard (yes/no) [y]: 2012 Aug 26
19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 1 ok (Serial
number DTM1437011H)
2012 Aug 26 19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power
supply 1 ok
2012 Aug 26 19:09:15 switch %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 1
detected (Serial number JAF1524AKTF)
2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 3
detected (Serial number JAF1448BPGN) Module-Type 10 Gbps Ethernet Module
Model N7K-M132XP-12
2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 3
powered up (Serial number JAF1448BPGN)
2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 5
detected (Serial number JAF1448BBDK) Module-Type 10/100/1000 Mbps
Ethernet Module Model N7K-M148GT-11
2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 5
powered up (Serial number JAF1448BBDK)
This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Hostnames are configured using the switchname or hostname command. Initially the hostname
command was not available when NX-OS was first forked from SAN-OS. Since this caused a lot of
confusion the hostname command has been ported back into NX-OS. When configured, the one that is
last used will be shown in the configuration. Pay attention to this as this might be a requirement during
the lab.
switch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Configuring the domain-name and DNS lookups is equal as to how it’s done within IOS.
To ensure both SSH and Telnet connections are allowed towards the management (mgmt0) interface of
the Nexus you should enable Telnet using the feature command. SSH is the default mechanism to
manage the Nexus switch and therefore the Telnet feature needs to be re-enabled. By default the switch
rejects any attempts of telnet.
To save your configuration using the wr command. The only option is to create a CLI alias, which can be
executed from anywhere in the CLI. The write command is reserved for the erasing of the configuration
as we did at the beginning of the lab, so we need to use an abbreviation for that (what most engineers
use in real-life as well).
version 5.1(5)
interface mgmt0
ip address 10.160.48.241/24
Note: The management IP address wasn’t erased during the erase of the configuration. This doesn’t
mean you can just erase your configuration and reload from the Ethernet management connection,
because you are required to set a password for the admin user. Meaning you can’t log-in using the SSH
connection that is by default enabled.
Only after enabling the Telnet feature on the CLI you are able to establish a Telnet connection to the
device.
telnet> q
Connection closed.
IPexpert:~ rick$
Configure the other tasks by just navigating through the configuration. You will never remember all
options available within the OS, but by just navigating using keywords you should be seeing a lot of hints
pointing to the right solution according to what was asked in the task.
version 5.1(5)
interface mgmt0
no cdp enable
ip address 10.160.48.241/24
version 5.1(5)
interface mgmt0
no cdp enable
ip address 10.160.48.241/24
rate-limit cpu direction both pps 999 action log
version 5.1(5)
interface Ethernet3/1
SW1-1(config-if)# switchport
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# sw trunk allowed vlan 100-499
This will cause VLANS to be overwritten. Continue anyway? [yes] yes
Because this command caused a lot of outages in the world, Cisco implemented a security to ask you if
you are really sure that you want this command to be entered.
version 5.1(5)
interface Ethernet3/1
switchport
switchport mode trunk
switchport trunk allowed vlan 100-499
In Classic IOS you only saw the shutdown command under the interface. Within NX-OS the opposite is
true and you only see the no shutdown. Don’t think the port is enabled when you see this
configuration. You really need to see the no shutdown command under the interface before it really
works!
SW1-1(config-if)# no shut
SW1-1(config-if)# sh run int e3/1
version 5.1(5)
interface Ethernet3/1
switchport
switchport mode trunk
switchport trunk allowed vlan 100-499
no shutdown
SW1-1(config-if)#
To remove a specific range (or one VLAN) from the allowed range, you don’t need to enter the whole
range again as you would need to in older IOS releases. NX-OS supports the remove command and an
except command.
Be aware when using the except command, because this will remove all VLANs from the trunk without
warning you! Besides it’s not what the task asked you. You should remove the VLAN from the allowed
range without specifying it. We do this with the remove command.
version 5.1(5)
interface Ethernet3/1
switchport
switchport mode trunk
switchport trunk allowed vlan 1-332,334-3967,4048-4093
no shutdown
version 5.1(5)
interface Ethernet3/1
switchport
switchport mode trunk
switchport trunk allowed vlan 100-332,334-499
no shutdown
SW1-1(config-if)#
Of course the proctor will never be able to verify if you used this single command, but this workbook
prepares you for the lab and every possible question. So this task shows you the possibilities of the OS.
The next tasks will be about configuring the VTP protocol. This protocol ensures all VLANs are
distributed throughout the network. As with almost all features within NX-OS you need to enable it first
before configuring it. The tasks in the workbook will push you to configure about every feature VTP has
and the following configuration will show you all those steps.
If you check the Cisco Documentation about Layer 2 Switching you will see similar configuration
examples which is also what you will see when you sit the lab. Most questions can be related to the
Documentation as that was used during the creation of the lab itself.
Once enabled the VTP command becomes available. When you check the options
you have, you will see about every task can be answered with these
commands.
SW1-1(config)# vtp ?
domain Set the name of the VTP administrative domain
file Set the name of the VTP file name
mode Configure VTP device mode
password Set the password for the VTP administrative domain
pruning Set the adminstrative domain to permit pruning
VTP pruning is the option to allow VLANs to be removed from switches where they are not required.
This limits the broadcast domains created (automatically) using VTP. This is also the main reason why
VTP is not recommended to be used in large datacenters.
The task asks you to configure the latest version. Which is the highest number?
Vlan.dat has historically been the VLAN database. Today it’s the VTP database which is separately stored
on the flash. When you erase the configuration of a NX-OS device, the VTP information will remain on
the flash and will be configured again when re-enabled.
The mode we are looking for in the task is that SW1 is the VTP server for the network, while SW2 and
SW3 are the clients. This way SW2 and SW3 are not able to create vlans. When you try to do this, you
will get an error message.
MD5 Digest : 0xF3 0xA6 0x62 0x7A 0xB1 0x85 0x77 0x40
Configuration last modified by 0.0.0.0 at 8-26-12 21:14:51
Local updater ID is 0.0.0.0
VTP version running : 2
After creating the VLANs in the next task you should verify on all switches if VTP pushed the information
to those switches. After configuring the names of the VLANs, verify again, as VTP should be pushing that
information towards all switches as well.
--------------------------------------------------------------------------
-----
As you can see the name of VLAN 104 hasn’t been changed yet. This is caused by the fact that VLANs are
still located in some sort of database within the configuration. You first need to exit out of the VLAN
configuration before the changes are applied
SW1-1(config-vlan)# exit
SW1-1(config)# sh vlan
SW1-1(config)#
Now verify the VLANs on all 3 switches to ensure that everything has been synchronized using VTP.
The last option to configure in this task is a special case. You really need to pay attention to the fact
what is happening in the show command.
Pay close attention that in the IGMP snooping show command VLAN 105 is shown, while in the show
vlan, you don’t see this 105. You probably need to check the Cisco Documentation for this feature, but
it’s meant for ‘pre-staging’ VLANs without creating them first.
This means it’s possible to enable IGMP snooping on a VLAN before it’s created and applying this
configuration immediately when deployed.
It’s a corner-case feature, but Cisco is known to use those particular features in 1 or 2 point tasks within
the CCIE lab. So this could be easy points to get when you know what they are looking for, or you could
easily loose points when you don’t and also have no clue where to look for. Again looking at Cisco
Documentation helps a lot in this case and it should be easy to spot this feature in the layer 2
documentation guide.
Another feature to configure is a service-policy in this section. With this policy you can apply QoS
markings or Policing at VLAN level instead of port-level. Be aware that you will mark and police ALL
traffic entering and exciting the VLAN on this switch.
After enabling the IGMP snooping this shows up in the show output of IGMP snooping just like the task
required you to. Still the VLAN hasn’t been created yet.
SW1-1(config)# exit
SW1-1# wr
[########################################] 100%
Copy complete, now saving to disk (please wait)...
SW1-1#
Save your configuration before moving on with the next task. Frequently saving you configuration might
save you a lot of trouble when you are configuring features that might cause the box to crash. Or you
reload the box without saving and you lose precious points!
Private VLANs are set-up in such a way that traffic is protected between certain ports. There is a Primary
VLAN which only contains the so-called Promiscuous ports. These ports can receive all traffic from all
ports that are associated with this primary VLAN. This means that the ‘client’ ports are in other VLANs
than the promiscuous port.
The secondary VLANs can be create in 2 ways. The “Isolated” and “Community” VLANs are these 2 types
of secondary VLANs. The Isolated version means that all ports that are configured in these VLANs cannot
communicate to each other. As pointed out in the drawing below. Within the layer 2 domain of this
particular VLAN ports will never communicate with each other, but can only talk to the promiscuous
port in the primary VLAN which they are associated with. The second type is the Community VLANs. All
interfaces in these VLANs can communicate to each other within the same Community VLAN. This
means that hosts in different community VLANs cannot communicate to each other. Again these
community VLANs are able to talk to the promiscuous port in the primary VLAN which they are
associated with.
The drawing below illustrates the possible (and impossible) traffic flows.
The associations / mappings between primary and secondary VLANs need to be manually configured on
the switch ports. The final step is that it’s also possible to trunk Private VLANs between different
switches. This means that a promiscuous port doesn’t need to be on the same switch as the host ports.
Therefore a whole isolated section of your network can be created on different switches.
The promiscuous port doesn’t have to be a fixed Ethernet port. It’s also possible to create a VLAN
interface (SVI) which is the promiscuous port for that particular Primary VLAN.
The first assignment point you to create a promiscuous port for an external firewall which will receive all
traffic connected to Primary VLAN 200.
As with all features within the NX-OS platform the private VLAN feature also needs to be activated first,
before any commands are available within the CLI.
Then the VLAN needs to be created and the VLAN needs to be made primary. Which we do under the
VLAN configuration.
Still we don’t see anything, but this is due to the fact that we need to exit out of the VLAN configuration
before it’s applied to the switch.
SW1-1(config-vlan)# exit
ERROR: Private VLANs can only be configured in VTP transparent/off modes
We receive an error that VTP is not possible to carry private VLAN information. This is true for the
supported versions of VTP on the NX-OS platform. Older Catalyst switches running IOS or even CatOS
also support VTP version 3, which is capable of carrying the private VLAN configuration. This code is
currently not implemented in NX-OS.
When we change VTP to transparent mode we do see that the changes are applied and that we enabled
a primary VLAN. We don’t have to create any mappings yet as that will come in the following
assignments.
The next assignment is a combination of enabling isolated ports on the same switch. Before they are
able to communicate to the promiscuous port and the firewall some mappings need to be created.
After the creation of the new isolated VLAN a mapping needs to be made between the primary and
secondary VLANs. This is done under the primary VLAN configuration and under the promiscuous port.
This model of having to create the mapping on multiple places is sometimes a bit difficult to remember
every spot where the mapping needs to be done, but in real life you will see you get a lot of flexibility
with this model and you are able to create almost any topology with this flexibility.
Now the final step is to create the same mapping on the isolated host ports.
Don’t forget to enable the port on the Nexus 7000 as switchport, as by default the ports are in non
switching mode.
After the mapping has been created on all 3 ports (promiscuous and host ports) you will see this
mapping in the show vlan. Here all 3 ports are shown, but this doesn’t mean that ports can
communicate to each other. The type of ‘isolate’ means that this is not possible for the host ports
and only for the promiscuous port.
The following 2 assignments point you in the direction of community VLANs. These VLANs are able to
communicate to each other, but not between different community VLANs and of course they are able to
communicate towards the promiscuous port.
Remember that all traffic on this promiscuous port is untagged. So it’s not possible to distinguish traffic
between different secondary VLANs on the firewall (virtually connected to port Ethernet 3/19).
When the new community VLANs are added, again the mapping needs to be created between the
primary and the secondary VLAN.
The following is that the interfaces need to be assigned to the correct community.
Next are the second set of ports that are not allowed to communicate to hosts in VLAN 202
And finally don’t forget to add the community VLANs to the promiscuous port, otherwise they won’t be
able to communicate with the firewall!
Don’t forget the ‘add’ keyword in the command; otherwise you will overwrite all current enabled VLANs
just like it is possible with the trunk allowed vlan command!
The next task requires you to create a different primary VLAN. This primary VLAN will not be configured
towards a physical interface, but will be configured under a layer 3 interface on the core switch (SW1-1).
This is a virtual promiscuous port. Configuration however is basically the same as a physical interface!
On the NX-OS platform it’s not possible to configure VLAN interfaces (SVIs) right away. Again you need
to enable the feature first!
On the SVI the private VLAN mapping to the primary VLAN needs to be configured.
For the next task another isolated VLAN needs to be created. This VLAN is associated with the special
primary VLAN. After that configuration the mapping is visible on the SVI of VLAN 204.
The next task requires the use of so-called private VLAN trunk links as the hosts are not connected
directly to the SW1-1, but to SW2. Private-VLAN trunk links can carry traffic for normal VLANs and
private VLANs at the same time.
The configuration can be to transfer primary VLANs to multiple switches or to transfer secondary VLANs
across multiple switches. In this task we configure the first trunk link to pass only secondary VLANs.
SW1-1
SW1-1(config)# int e3/1
SW1-1(config-if)# sw mode private-vlan trunk ?
promiscuous Port mode trunk promiscuous
secondary Port mode trunk isolated
SW2
SW2(config)# feature private-vlan
SW2(config)# vtp mode transparant
SW2(config)# vlan 205
SW2(config-vlan)# private-vlan isolated
SW2(config-vlan)# exit
When configuring private VLAN trunks you don’t need to change the normal trunk allowed vlan
command, as the private VLANs on the private-vlan trunk allowed list are automatically
added to the normal VLAN allowed list. On SW2 the same configuration is required to transfer the
secondary VLAN.
As SW2 does not know VLAN 204 (the Primary VLAN) it’s not necessary to create the association,
because the mapping for this is done on SW1-1.
Next are the hosts that need to be configured in the newly created secondary VLAN on SW2. Remember
that VTP is disabled for this use so you need to manually create the private VLAN on SW2. The host
association does need to be made to ensure the interface is a member of VLAN 205.
Final task is that a second trunk link needs to be enabled to extend the current VLANs. You are required
to use a different trunk link for this use. There are 2 ways to solve this case. The easiest way is to just
extend the secondary VLANs, just like we did in the previous assignment. The other solution is to extend
the primary private VLAN to SW2 and create a new VLAN 201 and 202 to connect those special hosts.
The problem which is created then is that the community VLAN on SW2 is not the same as the
community VLAN on SW1, which means that your connectivity is broken and you are not completing the
requirements of this task successfully.
SW1-1
SW1-1(config)# int e3/2
SW1-1(config-if)# sw mode private-vlan trunk secondary
SW1-1(config-if)# sw private-vlan trunk allowed vlan 201-202
SW1-1(config-if)# sw privat association trunk 200 201
SW1-1(config-if)# sw privat association trunk 200 202
SW2
SW2(config)# vlan 201
SW2(config-vlan)# private-vlan isolated
SW2(config-vlan)# exit
Copyright © by IPexpert. All rights reserved. 40
CCIE Data Center Lab Preparation Detailed Solution Guide
One important difference with classical STP is that Rapid-PVST+ assumes that network (core) links are
configured point-to-point without any hops in between (hubs). This assumption ensures a much quicker
convergence time and is interrupt driven instead of timer driven like in classical STP.
The spanning-tree port type is an important command which you should configure wisely! It can help
you convergence both for host and network links a lot when this is properly configured. When host ports
are configured as Edge ports they are also not able to start a Topology Change (TC) towards the network
which initiates spanning-tree convergence. The flap of an edge interface therefore does not start a
topology change in the network. Keeping the spanning-tree network much more stable!
The first task tells you to configure all host connecting ports should be configured as ‘edge’ ports so they
aren’t generating topology changes. Do this by using a range command will ease up your life and save
you precious CCIE lab time.
SW2
SW2(config)# interface e1/10-15
SW2(config-if-range)# spanning-tree port type ?
edge Consider the interface as edge port (enable portfast)
network Consider the interface as inter-switch link
normal Consider the interface as normal spanning tree port
SW3
SW3(config)# interface e1/10-15
SW3(config-if-range)# spanning-tree port type edge
SW3(config-if-range)#
The command warns you that it will only have effect when the ports are configures in switchport
mode access plus that spanning-tree portfast is enabled so interfaces will come up fast when hosts
are connected.
The second assignment requires you to configure a root configuration for the spanning-tree instance in
VLAN 101. There are 2 ways to do this, but there is no indication which one needs to be used. Then use
whatever you think is your favorite way to configure this.
The first way is by configuring manual priority values. Choose low values for this, just as a best practice.
There are fixed values that you are able to use for configuring the spanning-tree priority. Don’t mind if
you don’t reminder these from the top of your head. When you type it in wrong, the CLI will
automatically tell you which values are possible to fill-in.
The next, and currently preferred, way is by enabling the root functionality by using the dedicated
commands for this. This means that the switch will configure a dedicated priority for both the root and
backup root bridge. The default root priority value is 25476. When there is already a bridge on the
network with a lower priority than this (meaning it’s the current spanning-tree root bridge) the switch
will choose a priority value of 4096 lower than the current lowest priority value in the network, so
when using this command the switch will always become the root bridge. Of course this is not
Copyright © by IPexpert. All rights reserved. 42
CCIE Data Center Lab Preparation Detailed Solution Guide
maintained and when a new switch with a lower value comes online, that will become the root bridge.
This is a single configuration task so at the moment you enter the command, the switch will become the
root.
For the backup root bridge this fixed value is 28672 without a fallback mechanism to ensure it’s always
the backup root. The switch can’t determine the backup root bridge priority value as this is unknown on
the local switch.
The next assignment tells you to ensure that the timers of the Rapid-PVST+ process for VLAN 101. You
can easily configure lower timers for this, but you will you know what the best timer values for this
network are? Well that’s quite difficult. Therefore NX-OS has an automatic way of configuring this by
using the special ‘diameter’ command. This diameter command automatically calculates the best
timer values based on the maximum amount of hops between 2 hosts of the network. As our network
consists of 3 layer 2 switches, so the diameter should be configured on 3.
SW2
SW2(config)# spanning-tree vlan 101 root primary diameter 3
SW3
SW3(config)# spanning-tree vlan 101 root secondary diameter 3
For assignment 4 and 5 of this task it’s required to read these together, otherwise you might be
configuring the same thing twice, which will cost you time. The task is that SW1 is the root bridge for
VLAN 102. Together with that, any new (unknown) switch may become the root bridge of the network.
Of course you will never know what the pre-configuration of that new switch might look like, but we can
take some assumptions here and ensure that the priority values are as low as possible. The key point in
assignment 5 is that the switch will be configured with default spanning-tree priority values. The default
spanning-tree priority value is 32768, which means we need to configure our network for values lower
than that and ensuring SW1 will be the primary root bridge for VLAN 102.
SW1
SW1-1(config)# spanning-tree vlan 101 priority 8192
SW2
SW2(config)# spanning-tree vlan 101 priority 16384
SW3
SW3(config)# spanning-tree vlan 101 priority 16384
The next assignment is about steering traffic within the spanning-tree network. This means that we are
going to change spanning-tree costs to alter the layer 2 path through the network and influence which
link is going to be blocked. By default Rapid-PVST+ on NX-OS is using the so called short path-cost
Copyright © by IPexpert. All rights reserved. 43
CCIE Data Center Lab Preparation Detailed Solution Guide
method. This means that a 16-bit number is used to determine the cost. You can change this so that a
32-bit number is used to calculate the cost.
Then links can be manually changed what cost is used. On trunk links this can be changed per VLAN. By
default all ports are set to auto. This means that according to the link bandwidth a specific cost is
attached. Depending on if short or long path-cost method is configured.
With a 16-bit number values of 1 up to 65,535 can be configured. The assignment requires you to
configure numbers of over 100,000. This means that all switches need to be configured with a long
path-cost method.
SW1
SW1-1(config)# spanning-tree pathcost method long
SW2
SW2(config)# spanning-tree pathcost method long
SW3
SW3(config)# spanning-tree pathcost method long
The next task is the true traffic steering / engineering assignment, where you are asked to ensure that
SW1 is always using SW3 to reach the root bridge of VLAN 101. We just configured SW2 to be the root
bridge of VLAN 101.
There are a number of things to take a look at in this position. First is that spanning-tree will always use
the shortest path towards the root bridge as primary path. Meaning that it will elect its root port on that
path. Because we have a loop in our network, there will be links in a blocking state. In the current design
when we take a look at root ports and costs. The links between SW1 and SW3 will become blocking as
the following drawing illustrates.
Now we need to change this behavior as we want all traffic to flow through SW3. This means that the
links between SW1 and SW2 need to be on blocking, as we do need a path to the root switch without
any loops. There are various ways in which you are able to achieve the same goal, but the easiest way in
this case is by raising the cost of the links from SW1 to SW2. All these links are are 10 Gigabit Ethernet
connections so the default cost of all links in the network is 2,000. We are required to change the cost
with values higher than 100,000. So let’s change them to 4,000,000 to ensure that no matter what kind
of connection is made between those switches, the default path cost method will never elect it as a
better path than the path between SW1, through SW3 to SW2.
The drawing above illustrates the changes and how traffic will flow towards the root bridge. The links
between SW1 and SW2 are now in blocking state to prevent loops in the network and we accomplished
the task!
The next assignment requires you that for all inter-switch-links the fourth link is used as the primary link.
All other links will become blocking. Cost is not an option, because the cost is based on bandwidth and
not on a port level. Spanning-tree knows another config knob to use in the case where there are
multiple equal bandwidth links between bridges like in our network. In normal networks you would use
port-channeling for this features, but in this case we have multiple ports, of which most will become
blocking. Either way we want to ensure that the fourth link will become primary.
This is done using the port-priority feature. As a tie-breaker when default port-priority of 128 is used the
port number is used. So in this case the lowest number wins, which is never the fourth link in our case.
We want to change the port-priority on the fourth link to a lower value than the default of 128. Be
aware that the port-priority can only be configured with increments of 32. Again just like the spanning-
tree root priority, the CLI will tell you which values are available for use in this case.
We will use priority 64 (any number lower than 128 is good enough) and place it on all ‘4th’ links
between the 3 switches in the topology.
SW1
SW1-1(config)# int e3/4
SW1-1(config-if)# spanning-tree port-priority ?
<0-224> Port priority in increments of 32
SW2
SW2(config-if)# interface e1/4
SW2(config-if)# spanning-tree port-priority 64
SW2(config-if)# interface e1/8
SW2(config-if)# spanning-tree port-priority 64
SW2(config-if)#
SW3
SW3(config-if)# interface e1/4
SW3(config-if)# spanning-tree port-priority 64
SW3(config-if)# interface e1/8
SW3(config-if)# spanning-tree port-priority 64
SW3(config-if)#
The next assignment asks you to configure the lowest convergence times for VLAN 103. Now we can’t
use the diameter command as that will not configure the lowest times. In this case the solution is
simple. Configure all spanning-tree timers to their lowest value possible.
SW1
SW1-1(config)# spanning-tree vlan 103 ?
<CR>
, Multi range separator
- Range separator
forward-time Set the forward delay for the spanning tree
hello-time Set the hello interval for the spanning tree
max-age Set the max age interval for the spanning tree
priority Set the bridge priority for the spanning tree
root Configure switch as root
SW2
SW3
SW3(config)# spanning-tree vlan 103 hello-time 1
SW3(config)# spanning-tree vlan 103 forward-time 4
SW3(config)# spanning-tree vlan 103 max-age 6
The next assignment is a little bit cryptical. This is mainly done to test your skills of finding examples in
the documentation very fast. When you look up the layer 2 documentation of NX-OS and search for
‘Rapid Connectivity’ there is only 1 hit. This is also the technology making Rapid-PVST+ really rapid. As it
will assume a point-to-point link infrastructure instead of a shared infrastructure (multiple layer 2
bridges on a single segment).
The answer is quite simple. Make every inter switch link a point-to-point link.
SW1
SW1-1(config)# int e3/1-8
SW1-1(config-if-range)# spanning-tree link-type point-to-point
SW2
SW2(config)# int e1/1-8
SW2(config-if-range)# spanning-tree link-type point-to-point
SW3
SW3(config)# int e1/1-8
SW3(config-if-range)# spanning-tree link-type point-to-point
The final assignment requires you to clear all spanning-tree configuration from the switches so you can
make a clean slate before moving on to the next section of this chapter
This causes a much higher scale as the control-plane overhead of this protocol is much less than the
Rapid-PVST+ protocol used in the previous task. However it has some downsides as that configuration of
the VLAN to instance mapping needs to be equal on all switches otherwise you will get serious outages
on your network. This requires good thinking about which VLAN is added where.
Copyright © by IPexpert. All rights reserved. 48
CCIE Data Center Lab Preparation Detailed Solution Guide
You will see that most assignments in this task are equal to the Rapid-PVST+ task, but require different
thinking, where you need to apply instances instead of VLANs.
First assignment asks you to configure MST (IEEE 802.1s) on all 3 switches. Then you need to configure
some parameters and ensure that the MST is working on all switches. What this means is that all
configuration like name, revision number and VLAN to instance mapping needs to be equal on all
switches in order for MST to work properly. Otherwise you will run into issues in where some traffic
might even be blocked.
SW1
SW1-1(config)# spanning-tree mode mst
SW1-1(config)# spanning-tree mst configuration
SW1-1(config-mst)# ?
abort Exit region configuration mode, aborting changes
instance Map vlans to an MST instance
name Set configuration name
no Negate a command or set its defaults
private-vlan Set private-vlan synchronization
revision Set configuration revision number
show Display region configurations
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW2
SW2(config)# spanning-tree mode mst
SW2(config)# spanning-tree mst configuration
SW2(config-mst)# name IPexpert
SW2(config-mst)# revision 5
SW2(config-mst)# instance 1 vlan 10-99
SW2(config-mst)# instance 2 vlan 100-199
SW2(config-mst)# instance 3 vlan 800-1299
SW3
SW3(config)# spanning-tree mode mst
SW3(config)# spanning-tree mst configuration
SW3(config-mst)# name IPexpert
SW3(config-mst)# revision 5
SW3(config-mst)# instance 1 vlan 10-99
SW3(config-mst)# instance 2 vlan 100-199
SW3(config-mst)# instance 3 vlan 800-1299
The next assignment requires you to use your imagination or check the documentation really quick.
There is a specific functionality under the same ‘spanning-tree mst configuration’
configuration knob where you can configure synchronization of private-vlans. This means that no matter
what number the secondary VLANs have, they are always mapped to the instance of the primary VLAN
to ensure an equal layer 2 topology for all private VLANs contained under 1 primary VLAN.
SW1
SW1-1(config)# spanning-tree mst configuration
SW1-1(config-mst)# private-vlan synchronize
SW2
SW2(config)# spanning-tree mst configuration
SW2(config-mst)# private-vlan synchronize
SW3
Verify that all configurations were done properly. Be aware that accept the VLANs that you configured.
All other VLANs (even when not created) are still mapped to instance 0. Therefore you should always be
aware of that (unconfigured) instance 0 when designing you spanning-tree topology.
The next assignment lets you configure SW2 to be the primary root bridge for instance 1. You should use
the lowest possible value, which is 0.
From SW2 it’s clearly visible that SW1-1 is the root bridge for instance 1.
The next assignment asks you to try and configure SW3 to be the root bridge for the MST instance 1
with use of the special root priority command. Why does this fail? Well that’s because what was
explained in the Rapid-PVST+ section. The command will (when it sees a lower priority root bridge
currently online) to lower the root priority with 4096 to ensure this switch becomes the root. The
problem is that SW2 is already using the lowest possible value. So this is not possible and the command
will be rejected.
Now the next task does not allow configuration on the switch itself which needs to be made backup root
bridge. This means we have to use the default configuration of spanning-tree. The default priority value
is 32,768. This means that when we increase the priority value of SW1 we are sure that SW3 will be
elected the new root bridge once SW2 fails.
Again we are asked to configure optimal network timers for this topology. The topology is still similar to
the one as used in the Rapid-PVST+ chapter, so we can use the diameter command here again! Pay
attention that changing other instances than instance 0 is possible for the timers or diameter
command, but this is useless as the timers can only be configured once. Timers are always applied for all
instances at the same time. Therefore the diameter command only works when configured for instance
0.
<output omitted>
Again we are required to use values over 100,000 when using traffic engineering / steering. So this
means that all switches need to be configured with a long path-cost method.
SW1
SW2
SW2(config)# spanning-tree pathcost method long
SW3
SW3(config)# spanning-tree pathcost method long
The traffic steering question is similar to the task before, but instead of steering a VLAN, a whole
instance can be steered. The command is slightly different. Again we use a cost of 4,000,000.
The next assignment requires a little bit of thinking. Like the previous task we were also asked to
configure port-priority to determine which local interface between switches will be used as the primary
one. Now with MST this is configurable per instance. As we have 4 instances we want to load share
among all interfaces. Configuring the same lower port-priority per instance on different interfaces does
the trick.
SW1
SW1-1(config)# int e3/1
SW1-1(config-if)# spanning-tree mst 0 port-priority 64
SW1-1(config)# int e3/2
SW1-1(config-if)# spanning-tree mst 1 port-priority 64
SW1-1(config)# int e3/3
SW1-1(config-if)# spanning-tree mst 2 port-priority 64
SW1-1(config)# int e3/4
SW1-1(config-if)# spanning-tree mst 3 port-priority 64
SW1-1(config)# int e3/5
SW1-1(config-if)# spanning-tree mst 0 port-priority 64
SW1-1(config)# int e3/6
SW1-1(config-if)# spanning-tree mst 1 port-priority 64
SW1-1(config)# int e3/7
SW1-1(config-if)# spanning-tree mst 2 port-priority 64
SW1-1(config)# int e3/8
SW1-1(config-if)# spanning-tree mst 3 port-priority 64
SW2
SW2(config)# int e1/1
SW2(config-if)# spanning-tree mst 0 port-priority 64
SW2(config)# int e1/2
SW2(config-if)# spanning-tree mst 1 port-priority 64
SW2(config)# int e1/3
SW2(config-if)# spanning-tree mst 2 port-priority 64
SW2(config)# int e1/4
SW2(config-if)# spanning-tree mst 3 port-priority 64
SW2(config)# int e1/5
SW2(config-if)# spanning-tree mst 0 port-priority 64
SW2(config)# int e1/6
SW2(config-if)# spanning-tree mst 1 port-priority 64
SW2(config)# int e1/7
SW2(config-if)# spanning-tree mst 2 port-priority 64
SW2(config)# int e1/8
SW2(config-if)# spanning-tree mst 3 port-priority 64
SW3
SW3(config)# int e1/1
SW3(config-if)# spanning-tree mst 0 port-priority 64
SW3(config)# int e1/2
SW3(config-if)# spanning-tree mst 1 port-priority 64
SW3(config)# int e1/3
SW3(config-if)# spanning-tree mst 2 port-priority 64
SW3(config)# int e1/4
SW3(config-if)# spanning-tree mst 3 port-priority 64
SW3(config)# int e1/5
SW3(config-if)# spanning-tree mst 0 port-priority 64
SW3(config)# int e1/6
SW3(config-if)# spanning-tree mst 1 port-priority 64
SW3(config)# int e1/7
SW3(config-if)# spanning-tree mst 2 port-priority 64
SW3(config)# int e1/8
SW3(config-if)# spanning-tree mst 3 port-priority 64
The final 2 assignments require a little digging in the documentation again. These are exotic features
that are easily found when you have the documentation open.
SW1
SW1-1(config)# spanning-tree mst max-hops 10
SW2
SW2(config)# spanning-tree mst max-hops 10
SW3
SW3(config)# spanning-tree mst max-hops 10
The final in this task is to enable a pre-standard version of MST on a specific interface. Something rarely
seen in production, but it’s possible in this version of software so it’s better that you are aware that it’s
possible.
With these kinds of features it’s highly recommended to use the documentation. As most of the times
the question, or specific keywords in it, are already mentioned somewhere in the documentation. Then
they are easily spotted and configured in a few minutes.
So opening up the Cisco Documentation for layer 2 features is highly recommended as that makes
searching a lot faster.
The first assignment points you to the direction of spanning-tree port types. As mentioned in the
spanning-tree chapters this is very relevant for your spanning-tree design. When ports are configured
for Edge functionality they will never trigger a topology change in the network. Within the chapter you
were asked to configure certain ports. It’s also possible to configure all unconfigured ports for a
dedicated use.
Next are the special spanning-tree features. The first asks to ensure a port is shut down when BPDUs are
received. This is directly related to the BPDU guard feature.
The next feature filters BPDUs from edge interfaces or trunk interfaces. Be aware when you configure
this on your production network as this might cause bridging loops within your network!
The next assignment asks you to configure the BPDU filtering on an interface on SW2. The problem is
that we can’t configure the command directly under the interface. This means that we will be
configuring the feature globally for every interface (unconfigured). One thing is that we will be
configuring all interfaces, but including Ethernet 1/10 and that means we accomplished this task.
The question 5 relates to the root guard feature. Usually configured on edge ports. Meaning that it will
protect you against that new root bridges are connected to that particular interface. The port will go in
err-disable state when this occurs.
Before the root guard feature works, you first need to re-enable BPDU processing on the interface,
otherwise the superior BPDU’s are already caught in the BPDU filter and never processed. The
spanning-tree bpdufilter disable command will not work in this case as.
The next task is also a guard feature. Instead of making the port a root port. You can also guard a port
for becoming a designated port. Again we first need to disable the BPDU filter feature, before the loop
guard feature will work.
The next feature is a special case feature where you would always need the CLI help or the Cisco
Documentation. The command disables the Rapid-PVST+ interoperability when MST is enabled.
The final 2 questions in this task point you in the direction UDLD or Unidirectional Link Detection. This is
a Cisco proprietary protocol where it becomes possible to detect a single sided link failure, both on fiber
optic and copper cables. The drawing below illustrates what is meant with this.
The protocol can be enabled in 2 ways. Normal or Aggressive mode. The aggressive mode has the
advantage of faster timers and this ensures that traffic is being discarded.
The architecture is so that the FEX (short for Fabric EXtender) is configured through the parent switch
which can be either an Nexus 7000 or an Nexus 5000 switch, but not mixed at the same time. It’s
possible with Virtual Port Channels (vPC), which is tested in a later chapter, to connect a FEX to 2 parent
switches and still manage them as a remote line card. However this configuration is tested in the vPC
related chapter.
Current we will be configuring the FEX only to SW2 so we get the following architecture in the network.
The configuration for the fabric extenders is relatively simple. This task requires you to match the
output. What we can read from the output and the questions is that we need to configure the FEX with
a special name, we need to turn on the beacon LED on the FEX and we need to assign the FEX number of
101 to this particular FEX.
The FEX number is also the linecard number which you will be configuring the host interfaces with. The
FEX number (or chassis ID) needs to be between 100 and 199 and can be pre-staged. So when
connecting certain FEX switches to the Nexus 7000 or 5000 they will automatically get the right FEX
number associated. This is based on the FEX serial number. When no staging is done, the FEX number is
associated to the FEX when it’s detected on the interface when it’s enabled.
What’s also shown in the output is that the FEX is connected through port-channel 4. Now port-
channeling will be tested in detail in a later chapter. The purpose in this task is to build a port-channel
where the FEX is connected to. You can do this by just configuring 1 interface in a port-channel and let
the FEX connect to that. Which in that case turns out that the FEX is connected to that port-channel.
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW2(config)#
SW2(config)# interface port-channel 4
SW2(config-if)# switchport mode fex-fabric
SW2(config-if)# fex associate 101
SW2(config-if)# no shutdown
Task 7 is completed
The port-profiles are typically used in the Nexus 1000V where virtual ethernet interfaces are created
once VMs come online, so the configuration is always based on a template. Within the other Nexus
platforms it’s also possible to configure these templates for any kind of interface, both physical and
virtual. Then you apply this template to one or more physical / virtual interfaces of the kind of port-
profile that you made.
This can save you a lot of work in a production environment too. The configuration is straight forward.
The configuration tasks are based on miscellaneous features that can be enabled on interfaces. First is
flowcontrol, which can be activated both in send and receive direction. The straight/cross cable
detection only works on copper interfaces and is called Auto MDIX. Finally the usage statistics on
show interface are configured using the load-interval command. Since NX-OS you can configure
up to 3 counters to be more granular in the averages that are measured.
As previously mentioned, it’s possible to create different types of port-profiles dependent on the type of
interface that you are configuring with it.
Next up is configuring the features under the port profile. The port-profile is configured like a regular
interface with the same options.
The load-interval is different from IOS. When the command is configured without specifying the counter
it’s configuring the normal IOS counter, where only 1 average delay timer can be configured. In NX-OS
you can configure up to 3 counters to measure different averages. Counter 1 is the default counter and
numbers 2 and 3 are specifically mentioned in the show interface output.
SW1-1(config-port-prof)# load-interval ?
<30-300> Load interval delay in seconds
One important note for port-profiles is that you should never forget the state enabled command.
Without this command the port-profile will never be active or will be enabled, so interfaces will not
come up with the right configuration without this config knob.
SW1-1(config-port-prof)#
SW1-1(config-port-prof)# sh run port-profile
version 5.1(5)
port-profile type ethernet IPEXPERT
switchport
switchport mode trunk
switchport trunk allowed vlan 101-104
flowcontrol receive on
no mdix auto
load-interval counter 1 30
load-interval counter 2 60
load-interval counter 3 120
no shutdown
state enabled
version 5.1(5)
interface Ethernet5/16
inherit port-profile IPEXPERT
After applying the port-profile to the interfaces, meaning that the configuration is only visible once in
the configuration, the interface configuration only shows the application of the port-profile to the
interface.
When verifying the interface. All configuration from the port profile is visible, like the Auto MDIX, the
3 load interval counters and the layer 2 trunk mode.
Now you have successfully completed task 8 and chapter 1 of this workbook. Take a little break before
continuing with configuring the layer 3 routing features!
Chapter 3: Data
Center Networking
Layer 3
Infrastructure (NX-
OS)
Data Center Networking Layer 3 Infrastructure is intended to let you be familiar with the NX-OS Layer 3
features on the Nexus platforms to create a basic routed network. We highly recommend creating your
own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much
easier when you step into the real lab. The lab is divided in two pieces. During the first tasks you will be
configuring a dynamically routed layer 3 network using EIGRP and OSPF protocols. The second part of
this chapter is based on the Cisco proprietary technologies FabricPath and OTV. Multiple topology
drawings are available for this chapter.
General Rules
• Try to diagram out the task. Draw your own connections the way you like it
• Take a very close read of the tasks to ensure you don’t miss any points during grading!
• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Solutions
During the initial configuration there are VDC’s configured on the Nexus 7000 switch (SW1-1). This
configuration is then used throughout the first couple tasks of this chapter. You can log-in to the VDCs
using the command: switchto vdc <vdcname>. To switch between VDCs use the same command.
To go back to the default VDC you can use: switchback.
SW1-1# switch
switchback switchto
SW1-1# switchto ?
vdc Manage Virtual Device Context
SW1-1# switchback ?
<CR>
There is not really a best way to configure this network. As long as it works and within the requirements
of the chapter the solution is good. The following configuration is considered best practice in this case.
The first assignment is to ensure that all relevant switches carry VLANs that are required for those
switches. The topology is designed to be built using Switched Virtual Interfaces (SVIs) also called VLAN
interfaces. The SVI’s are created as Layer 3 end of the layer 2 domain and are very common in
Datacenter environments. We are using the SVI’s in this topology as it saves a lot of physical cables and
ensures a very flexible environment where we can add and remove certain hosts from the interfaces.
Along with the creation of the VLANs and the IP addressing you also need to be sure that the VLANs are
transported between the switches that are used as end-point of the interface.
Within the VDC configuration you are able to see the physical interfaces that are connected between
the different logical switches. On those interfaces a trunk connection carrying the specific VLANs should
be configured. Of course you could enable all VLANs on all interfaces, but this is not helping the topology
as spanning-tree will kick in and block at least one connection for each VLAN somewhere in the topology
where (without any other spanning-tree configuration) you aren’t sure about which link will be blocked
for this.
Now we decide to only configure all VLANs that also have an SVI configured on the same switch. Along
with that only a single or a few VLAN(s) is/are allowed on the trunk link between the 2 switches that are
configured. Thus we are sure that never a loop exists in the layer 2 network. Which in this case means
that spanning-tree will never block any link in our network and we are running a perfect layer 3 network.
The IP addressing is stated in the drawing and we use the VLAN number as subnet ID and the address
noted with the switches (according to hostname) as the host portion of the address. All subnets are /24
or 255.255.255.0.
SW1-1
SW1-1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1-1(config)# feature interface-vlan
As we start with a new configuration we need to enable the SVI feature again.
We need to create the layer 2 VLANs that are used on this particular switch.
Connection to SW1-2
Connection to SW1-3
Connection to SW2
Now to verify everything is configured properly a show ip interface brief is enough to show you all the
details that we just configured.
The first switch is configured. Now finishing up by configuring all other layer 2 and layer 3 switches in
the network. Both Nexus 5500 switches contain the Layer 3 module so have the possibility to create a
full layer 3 topology.
SW1-2
SW1-2# conf t
SW1-2(config)# feature interface-vlan
Connection to SW1-1
Connection to SW1-4
Connection to SW3
Verification
SW1-3
SW1-3# conf t
SW1-3(config)# feature interface-vlan
Connection to SW1-1
Connection to SW1-4
Verification
SW1-4
SW1-4# conf t
SW1-4(config)# feature interface-vlan
Connection to SW1-3
Connection to SW1-4
Verification
SW2
SW2# conf t
SW2(config)# feature interface-vlan
Connection to SW1-1
Connection to SW3
This will be our only Layer 3 interface in the network without SVI. Still a VLAN is configured for this link.
This is configured using a sub-interface on the physical interface. We first make the interface a layer 3
port and then create the sub-interface while specifying the VLAN number using the encapsulation
command.
Verification
SW3
SW3# conf t
SW3(config)# feature interface-vlan
Connection to SW1-1
Connection to SW3
Verification
The final assignment is to create Loopback interfaces on the routers in the network so they can be
advertised. Loopback interfaces are typically used for managing a device in the network. Now with NX-
OS this is usually done using the mgmt0 interface on the devices. The best practice is indeed to use this
mgmt0 interface, instead you have a strict requirement to have a redundant management interface.
The mgmt0 interface is always non-redundant, so when this dedicated piece of hardware fails, you can’t
manage the switch.
It’s highly recommended to do software upgrades through the mgmt0 interface as this interface has a
much higher connection which is not rate-limited to the control-plane. When copying software images
through an in-band management connection (over a Loopback, Layer 3 interface or SVI) you will see
much lower transfer rates as this connection to the control-plane is rate-limited.
In this lab the Loopback interfaces will be used for various purposes and are a good test to see if all
routers are in the routed network. Don’t think you already passed a ‘full reachability’ statement yet,
because you could be missing some crucial links in the routing protocol which means you didn’t pass the
‘full reachability’ and loose precious points.
Now to finish the task, we create all Loopback interfaces on the switches.
SW1-1
SW1-1(config-if)# int Loopback 0
For once, it’s not necessary to enable this functionality within NX-OS!
After the creation of the Loopback interface we can verify it’s working properly with a show
interface or again a show ip interface brief.
SW1-1(config-if)#
SW1-2
SW1-2(config-if)# int Loopback 0
SW1-2(config-if)# ip add 198.18.0.12/32
SW1-3
SW1-3(config-if)# int Loopback 0
SW1-3(config-if)# ip add 198.18.0.13/32
SW1-4
SW1-4(config-if)# int Loopback 0
SW1-4(config-if)# ip add 198.18.0.14/32
SW2
SW2(config-if)# int Loopback 0
SW2(config-if)# ip add 198.18.0.2/32
SW3
SW3(config-if)# int Loopback 0
SW3(config-if)# ip add 198.18.0.3/32
The initial assignment asks you to ensure that SW1-3 can ping the loopback address of SW1-4. We can
configure a static route for this. A static route is created using a few keywords:
• IP Prefix
• Next-hop interface
• Next-hop IP address
• Properties
The network mask can be noted in classical dotted decimal notation (255.255.255.0) like in Classical
IOS or could be using the / notation (x.x.x.x/24) which means the same, but is much easier to
configure and easier to see how large the subnet mask is.
Next can either be directly the next-hop IP address where the router will select the next-hop interface
by itself by looking into the routing table for a connected prefix for this next-hop. On certain platforms
this next-hop can also be ‘recursive’, meaning it can point to another routing entry which in that
case points to another routing-entry, until it finds a directly connected prefix which is an
interface connected to the router. It depends very much on the platform how many levels of recursion
are supported. Best practice is to create static routes to point directly to a connected interface.
Instead of a next-hop IP address you can also specify a next-hop interface. Which usually points to a
point-to-point (serial or tunnel) interface which means that traffic that enters, needs to exit out on the
other end, so a next-hop IP address is not necessary. You can’t configure an Ethernet interface (physical
or SVI) as the next-hop interface, after that you always need to specify the next-hop interface. This is
because an Ethernet interface is a multi-access network. Meaning that multiple hosts reside on this
interface and the router will then never know where to forward the traffic to. Therefore it’s always
required to specify this destination IP address along with the interface. Now the router will not do a
look-up in its routing table to find the interface to send it out, as you just specified it.
After these items to configure the static route, a few properties can be configured which will be
discussed later in this chapter.
SW1-1(config)# ip route ?
A.B.C.D IP prefix in format i.i.i.i
A.B.C.D/LEN IP prefix and network mask length in format x.x.x.x/m
The destination prefix is the Loopback interface of SW1-4 and the next-hop address is the directly
connected SVI interface where the next-hop is the host on the other-end (SW1-4).
The second part of the question states that we should be able to ping it from SW1-3’s loopback address.
Now if we try that. This doesn’t work!
Pay attention that in classical IOS you were able to use the interface name as a source in this
command, but within NX-OS you always need to specify the source IP address instead of the
interface.
Why doesn’t the ping success? Well that’s because SW1-4 has no routing entry for the Loopback address
of SW1-3 in its routing table. Therefore a static route also needs to be configured on SW1-4 for the
return traffic back to SW1-3.
After configuring this static route on SW1-4. Both switches are able to ping each other’s Loopbacks while
using their own Loopback address as source for the traffic.
Next up in the second assignment of this task is to configure a static route so that SW1-1 and SW1-2 can
ping each other’s loopback addresses, but instead of using the directly connected links between those 2
switches they should travel across SW1-3 and SW1-4. This means that we need to point the static routes
over the correct interfaces.
SW1-1
SW1-1(config)# ip route 198.18.0.12/32 198.18.113.13
SW1-2
SW1-2(config)# ip route 198.18.0.11/32 198.18.124.14
Now the correct static routes are created, but a ping won’t succeed.
SW1-3
SW1-3(config)# ip route 198.18.0.12/32 198.18.134.14
SW1-3(config)# ip route 198.18.0.11/32 198.18.113.11
SW1-4
SW1-4(config)# ip route 198.18.0.12/32 198.18.124.12
SW1-4(config)# ip route 198.18.0.11/32 198.18.134.13
After configuring those additional static routes the ping from both switches succeeds, even while using
the local Loopback interface as the source address.
The next assignment is to configure some properties along with the static route. Besides that we need to
configure a black hole for a specific subnet. This means that we point the static route not towards an
interface or to a next-hop IP address, but towards a special interface. This interface is called the null0
interface. In other words: bit bucket. Every packet that is sent to that interface will be dropped and not
processed further.
Besides pointing the static towards the bit-bucket it also needs to have some properties attached, which
are easy to spot with use of the question mark in the CLI.
The only tricky part is that you need to know the default route preference or administrative distance on
NX-OS of static routes as you need to increase it with +1.
After typing the question mark menu, you can already see the default route preference value (in
classical IOS the administrative distance). This should then be upped with 1 to comply with the question.
Now it’s possible to configure a name for the static route, but for some reason it’s then impossible to
change the route preference again. There is no specific reason why this isn’t possible, this is a limitation
in the current CLI. The name feature has just been released, so the CLI is not totally good with it. These
are things you will see more common in NX-OS, where new features get introduced very frequently. The
OS is still growing in features, where Classical IOS has grown for the last 20 years, therefore some
features are not mature yet.
After configuring the right static route it can be verified using a show ip route where it’s shown that the
route preference is now 2 instead of 1 and that the static route is tagged with a value of 666. Route
tagging is used during redistribution to recognize certain routes in your topology as those tags are also
preserved through dynamic routing protocols.
SW1-2(config)# sh ip route
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
Verify the configuration using a show command. Pay attention that you need to use quotes in NX-OS
while doing a search with multiple words.
Now by default traffic that is dropped by the switch will send ICMP unreachable messages back to the
sender. These messages can be used by hackers to detect whether a certain destination is reachable or
not. In Classical IOS it was possible to configure this on the Null0 interface to generally disable it for the
dropped traffic through the Null0 interface, but in NX-OS this is only possible to configure on Layer 3
routed interfaces. Besides that a difference is made for ICMP unreachables and port-unreachables. The
question clearly states that the interfaces should not send out any unreachable message, so don’t forget
to add this one as well. SW1-2 has 5 VLAN interfaces (SVIs) so they should all be configured to not send
out the unreachable messages.
SW1-2
SW1-2(config)# int vlan 124
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
SW1-2(config-if)# int vlan 112
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
SW1-2(config-if)# int vlan 211
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
SW1-2(config-if)# int vlan 123
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
SW1-2(config-if)# int vlan 132
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
To finish the task remove all previously configured static routes from the configuration and continue
with the next task.
SW1-1
SW1-1(config)# no ip route 198.18.0.12/32 198.18.113.13
SW1-2
SW1-2(config)# no ip route 198.18.0.11/32 198.18.124.14
SW1-2(config)# no ip route 192.0.1.0/24 null 0
SW1-3
SW1-3(config)# no ip route 198.18.0.14/32 198.18.134.14
SW1-3(config)# no ip route 198.18.0.12/32 198.18.134.14
SW1-3(config)# no ip route 198.18.0.11/32 198.18.113.11
SW1-4
SW1-4(config)# no ip route 198.18.0.12/32 198.18.124.12
SW1-4(config)# no ip route 198.18.0.11/32 198.18.134.13
SW1-4(config)# no ip route 198.18.0.13/32 198.18.134.13
Task 3: EIGRP
The next task is about configuring the Enhanced Interior Gateway Routing Protocol or EIGRP. The EIGRP
protocol is the first dynamic routing protocol that is tested in the CCIE Data Center. The next task
regarding Open Shortest Path First (OSPF) is the second routing protocol. Other dynamic routing
protocols like RIP, IS-IS and BGP are not tested in this Data Center exam. Usually you will not find BGP
configuration in a typical Data Center environment on the Nexus switches, but that will be the task of
some kind of PE (Provider Edge) router that is above the core routing layer. This exam doesn’t focus on
that part and is only interested in the most widely used Interior Gateway Protocol like EIGRP and OSPF.
RIP is an outdated protocol which is used only in very special cases, whilst IS-IS is more a Service
Provider protocol, due to scaling advantages.
The EIGRP protocol is a Cisco proprietary protocol. Officially it’s an open document where it’s based on,
so every routing vendor should be able to write its own implementation, except nobody has. Therefore
EIGRP is considered Cisco proprietary.
The EIGRP protocol is a combination of Distance-Vector and Link-State protocols. It’s a very fast
converging protocol due to incremental updates sent to neighbors.
The DUAL (Diffusing Update Algorithm) calculates the best routes according to the shortest distance
(based on various link properties). When a successor route is chosen this distance combined with the
local link cost is then used as the Feasible Distance. This is the mechanism that makes EIGRP such a fast
converging protocol.
The DUAL does a second run through the routing table and calculates a second best path based on the
Feasible Distance. A route can become second best when that distance (advertised distance) is lower or
equal to the feasible distance. When this would not be the case and a second best path could be any
cost, you could run into routing loops as the router doesn’t know the exact topology like a link-state
protocol does, so the traffic could be crossing the same router again. A Feasible Successor is selected
and installed as a backup route to the Successor route. When the primary path fails the Feasible
Successor route is immediately installed as new best path and the router doesn’t have to calculate
another path.
Currently with initiatives like IP FastReRoute (FRR) or IP Loop-Free-Alternate (LFA) this mechanism is also
possible on OSPF and IS-IS, but EIGRP has been using this mechanism for a long time and is known to be
very successful in fast convergence!
The EIGRP metric / distance calculation is quite difficult. The full formula is as follows:
This calculation is not used in any network at this moment. This is because most of the K-values area at 0
and multiplication to 0 always result in a 0. Currently only the values Bandwidth and Delay (K1 and
K3) are used to calculate the metric of an interface in EIGRP.
With the introduction of Wide Metrics (64-bit metrics) additional K-values came into play. The first is
Jitter (in microseconds) and the second is Energy (in watts per kilobit). With this case you can enable
some sort of Performance Routing by using EIGRP metrics. A lower powered device should attract more
traffic than a high powered device for example.
There are a lot of scenarios in which this would be useful. By default the new K6 values are turned off in
the metric calculation. Though the values are added in the EIGRP update packets.
Wide metrics are also backwards compatible with regular metrics. When a non-wide-metric router
receives an update where the extended attributes are included it will transparently pass it through
without updating it, as a wide-metric-enabled router would do. Although it might result in suboptimal
routing in some cases.
One big advantage of EIGRP on NX-OS is that the auto-summary (classful routing) feature has been
disabled by default and is not possible to turn on as well.
The initial configuration of this task is to configure EIGRP on SW1-2 and SW1-4 whilst enabling loopbacks
in this network as well without forming adjacencies.
EIGRP uses an autonomous system number to identify if the routers will allow a neighbor relationship
with each other.
We first enable the EIGRP feature, which automatically enables our LAN_ENTERPRISE_SERVICES license
on the box. This particular test router has no license installed, so the CLI warns you about the grace
period which after the feature will be turned off. For testing purposes we don’t have anything to
worry about for now.
The only thing we need to configure for the process is to create the process with a name. We use
IPEXPERT in this question. The reason is that when the AS number is specified as the process name it
automatically gets this AS number assigned and the process is running.
Currently without specifying the AS number, the process is in a shutdown state as we can see in the
verification.
SW1-4(config-router)# sh ip eigrp
IP-EIGRP AS 0 ID 198.18.0.14 VRF default
Process-tag: IPEXPERT
Status: shutdown
Authentication mode: none
Authentication key-chain: none
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0
IP proto: 88 Multicast group: 224.0.0.10
Int distance: 90 Ext distance: 170
Max paths: 8
Number of EIGRP interfaces: 0 (0 loopbacks)
Number of EIGRP passive interfaces: 0
Number of EIGRP peers: 0
Graceful-Restart: Enabled
Stub-Routing: Disabled
NSF converge time limit/expiries: 120/0
NSF route-hold time limit/expiries: 240/0
NSF signal time limit/expiries: 20/0
Redistributed max-prefix: Disabled
SW1-4(config-router)# autonomous-system 64999
To get the process enabled we need to configure the AS number manually using the dedicated
command for this. Now when we verify, the process is running with the correct name and AS number
attached.
SW1-4(config-router)# sh ip eigrp
IP-EIGRP AS 64999 ID 198.18.0.14 VRF default
Process-tag: IPEXPERT
Status: running
Authentication mode: none
Authentication key-chain: none
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0
IP proto: 88 Multicast group: 224.0.0.10
Int distance: 90 Ext distance: 170
Max paths: 8
Number of EIGRP interfaces: 0 (0 loopbacks)
Number of EIGRP passive interfaces: 0
Number of EIGRP peers: 0
Graceful-Restart: Enabled
Stub-Routing: Disabled
NSF converge time limit/expiries: 120/0
NSF route-hold time limit/expiries: 240/0
NSF signal time limit/expiries: 20/0
Redistributed max-prefix: Disabled
SW1-4(config-router)#
The rest of the configuration is done under the interfaces that will be placed in the EIGRP process.
Within Classical IOS much more configuration was done under the process, but this has now moved to
the interface.
SW1-4
SW1-4(config-router)# int vlan124
SW1-4(config-if)# ip router eigrp IPEXPERT
SW1-2
SW1-2(config)# feature eigrp
SW1-2(config)# router eigrp IPEXPERT
SW1-2(config-router)# autonomous-system 64999
SW1-2(config-router)# exit
SW1-2(config)# int vlan124
SW1-2(config-if)# ip router eigrp IPEXPERT
SW1-2(config-if)#
Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately
comes up and is working correctly.
SW1-4
SW1-4(config-if)# sh ip eigrp nei
IP-EIGRP neighbors for process 64999 VRF default
SW1-2
SW1-2(config-if)# sh ip eigrp nei
IP-EIGRP neighbors for process 64999 VRF default
H Address Interface Hold Uptime SRTT RTO Q
Seq
(sec) (ms) Cnt
Num
0 198.18.124.14 Vlan124 12 00:00:16 10 200 0
2
SW1-2(config-if)#
Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately
comes up and is working correctly. Now without any advertised interfaces as it’s only enabled on a
point-to-point link.
Before we will advertise any subnets we first need to ensure that the adjacency between the
switches/routers is secure as stated in the question.
This is done using the authentication configuration keywords and a key chain with a valid key. It’s
possible to configure multiple keys with lifetimes so you can change your EIGRP authentication keys
without downtime by making the acceptance lifetime a little longer than the sending lifetime. That way
new keys are automatically used, but old keys are still accepted for a longer period so other routers get
time to also send this new key.
SW1-4
SW1-4(config)# key chain EIGRP_CHAIN
SW1-4(config-keychain)# key 1
SW1-4(config-keychain-key)# key-string IPEXPERT
As by the default parameters stated at the beginning of this workbook we will configure a password of
IPEXPERT to secure the adjacency.
The only possible authentication mode is md5. Simple-text / Clear-text authentication is not supported
in EIGRP.
version 5.0(3)
interface Vlan124
ip address 198.18.124.14/24
ip router eigrp IPEXPERT
ip authentication mode eigrp IPEXPERT md5
ip authentication key-chain eigrp IPEXPERT IPEXPERT
no shutdown
SW1-4(config-if)#
After all settings are configured we clearly see that the neighbor has gone down with the following
message:
SW1-4
2012 Sep 19 14:57:39 SW1-4 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT
[14861] (def
ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.12 (Vlan124) is down:
auth
entication mode changed
On the other switch the neighbor has also gone down with another message:
SW1-2
2012 Sep 19 14:57:39 SW1-2 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT
[14789] (def
ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.14 (Ethernet2/2) is
down: Auth
failure
Now we configure the other switch with the authentication settings to have both ends configured equal.
SW1-2
SW1-2(config)# key chain EIGRP_CHAIN
SW1-2(config-keychain)# key 1
SW1-2(config-keychain-key)# key-string IPEXPERT
Copyright © by IPexpert. All rights reserved. 86
CCIE Data Center Lab Preparation Detailed Solution Guide
After configuring the correct authentication settings the neighbor immediately comes back up and is
working as before:
SW1-2
SW1-2(config-if)# sh ip eigrp nei
IP-EIGRP neighbors for process 64999 VRF default
H Address Interface Hold Uptime SRTT RTO Q
Seq
(sec) (ms) Cnt
Num
0 198.18.124.14 Vlan124 14 00:00:05 28 200 0
6
SW1-2(config-if)#
SW1-4
SW1-4(config-if)# sh ip eigrp nei
IP-EIGRP neighbors for process 64999 VRF default
H Address Interface Hold Uptime SRTT RTO Q
Seq
(sec) (ms) Cnt
Num
0 198.18.124.12 Vlan124 11 00:00:37 7 200 0
6
SW1-4(config-if)#
Besides the securing of the adjacency we need to enable EIGRP on the Loopback interfaces so they can
be reached from both switches and no attempts to make adjacencies should be done on the Loopback
interfaces. This is done using the passive-interface command.
SW1-2
SW1-2(config)# int lo0
SW1-2(config-if)# ip router eigrp IPEXPERT
SW1-2(config-if)# ip passive-interface eigrp IPEXPERT
SW1-4
SW1-4(config)# int lo0
SW1-4(config-if)# ip router eigrp IPEXPERT
SW1-4(config-if)# ip passive-interface eigrp IPEXPERT
Now we are able to ping the loopback interfaces that are dynamically learned on SW1-2 and SW1-4.
• 198.18.4.0/24
• 198.18.5.0/24
• 198.18.6.0/24
• 198.18.7.0/24
• 00000100
• 00000101
• 00000110
• 00000111
Anything between the last 2 bits is changed in these static routes. So when we create a static route with
a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!
First we configure the static routes on SW1-4 pointing to null0 to ensure they are installed in the routing
table. When you create a static with a next-hop address which is not valid you will not see them being
installed in the routing table, except when they are marked with a permanent option, but this is not
available in NX-OS.
Next we configure the summary on the interface towards SW1-2 to ensure the statics are summarized
as one entry. Note that summary addresses get a default route preference of 5 in the routing
table.
Now when we check SW1-2 we still don’t see the summary address or any of the static routes appear in
the routing table.
We first need to ensure that the prefixes are injected in the EIGRP topology database by using a process
called Redistribution. This process ensures that prefixes from another protocol (or connected/static
routes) are injected the protocol where it’s configured. This can cause serious routing loops in the
network so beware when configuring this. As a pre-caution Cisco implemented a failure check that
redistribution always needs to be configured using a route-map to select which prefixes are entering the
redistribution process or not.
In our case we only need to configure these 4 static routes so additional filtering on the route-map is not
strictly necessary though highly recommended when this would be a real test. During the real test you
might find that a later task asks you to configure an additional static route which will then automatically
be redistributed in a protocol, which you probably don’t want. This might cause you loosing points in
both that new task and the older EIGRP task which you finished in the morning.
A route-map is required when configuring. You can not apply this command without specifying it
anymore.
Now when checking the routing table on SW1-2 we see the new summary address configured.
In the topology database of SW1-4 we see all 5 prefixes enabled, but only 1 is advertised to the other
routers. This is the advantage of distance-vector style protocols where you send your own routing table
to the neighbor and can therefore easily filter out certain prefixes that you want to hide.
Now we don’t comply with the question yet as we also want one specific prefix to be ‘leaked’ through
the summary. This is where we are going to use a leak-map to specify this single prefix to be leaked to
the other router (SW1-2).
Now after applying the leak-map we see the leaked prefix also in the routing table of SW1-2.
The rest of the questions in this task relate to certain EIGRP features. Again opening up the Unicast
Routing Configuration Guide really helps when solving these tasks as most of the question can relate to
something in this guide and walk you thru certain solutions.
The first is to enable wide metrics. Wide metrics are explained in the beginning of this task solution and
enable 64-bit wide metrics instead of 32-bit. These can be scaled down to 32-bit numbers again using a
scaling-factor. This factor can be adjusted along your preference which is also what the question states.
SW1-2
SW1-2(config-if)# router eigrp IPEXPERT
SW1-2(config-router)# metrics version 64bit
SW1-2(config-router)# metrics rib-scale 64
SW1-4
SW1-4(config-if)# router eigrp IPEXPERT
SW1-4(config-router)# metrics version 64bit
SW1-4(config-router)# metrics rib-scale 64
Next up is changing the EIGRP bandwidth utilization on an interface. This value can be changed. The
default is 50% of control-plane traffic, so lowering this with 10% brings it down to 40%.
SW1-2
SW1-2(config-if)# interface Vlan124
SW1-2(config-if)# ip bandwidth-percent eigrp IPEXPERT 40
SW1-4
SW1-4(config-if)# interface Vlan124
SW1-4(config-if)# ip bandwidth-percent eigrp IPEXPERT 40
Tuning the timers on an interface affects the neighborship. So beware of a flapping neighbor when
timers are changed. This is because the holddown timer is negotiated in the hello packets of EIGRP. The
question states to change the holddown timer to 4 times a hello packet.
By default hello packets are sent every 5 seconds and by default the holddown timer is set to 15
seconds, meaning it can miss 3 hello’s before declaring a neighbor down. So when we change SW1-2 to
a holddown timer of 20 seconds it should negotiate it down towards SW1-4 and from then on use the
new timer
Now we can verify this by issuing a show ip eigrp neighbor detail on the neighboring switch to verify it’s
using the new holddown timer.
SW1-4(config-if)#
Note that SW1-2 and SW1-4 are now using different values to declare each other down. This is exactly
what the question indicates.
The next 2 questions indicate some tuning to the EIGRP election process. The configuration guide is your
friend while answering these types of questions. The Stuck in Active timer and the Max-Hops
property needs to be changed to finish these questions and are easily found in the Tuning EIGRP
section of the configuration guide.
SW1-2
SW1-2(config-if)# router eigrp IPEXPERT
SW1-2(config-router)# timers active-time 5
SW1-2(config-router)# metric max-hops 50
SW1-4
SW1-4(config-if)# router eigrp IPEXPERT
SW1-4(config-router)# timers active-time 5
SW1-4(config-router)# metric max-hops 50
The final question of this task requires you to change value K3 on the interfaces. This value is used for
calculating the interface delay in the EIGRP metric formula:
The delay value is an administrative value on the interface, just like bandwidth and can be
administratively changed to do traffic engineering in EIGRP which is more recommended than to
changing the bandwidth value.
SW1-2
SW1-2(config-if)# interface Vlan124
SW1-2(config-if)# delay 500
SW1-4
SW1-4(config-if)# interface Vlan124
SW1-4(config-if)# delay 500
Task 4: OSPF
The next task is about configuring the Open Shortest Path First or OSPF. The OSPF protocol is considered
the most widely used dynamic routing protocol. It’s an IETF standard and every networking hardware
vendor has its own implementation of this protocol.
The current mostly used version and the version tested on this CCIE lab exam is OSPF version 2. This
version only support IP version 4 and has been out for many years. Currently there is a new version of
OSPF maturing which is Version 3. This version was initially developed solely for IP version 6, but has
now been extended so it also supports IP version 4. In the near future it’s expected that version 3 will
take over from version 2. The 2 versions are in no way compatible with each other, so you will have to
migrate your whole network over from version 2 to version 3. Some key features have changed in the
new version, particularly how interfaces/links are treated in the protocol.
This workbook focuses on the version used in the CCIE Data Center lab exam, which is OSPF version 2 for
IP version 4.
The OSPF protocol is a Link-State routing protocol. Which means that instead of pushing the routing
table to neighboring routers. The protocol will inform all neighboring routers within a single ‘area’ of the
state of each connecting link using link-state-advertisements (LSAs). This information is flooded
throughout the OSPF area to all routers. Once all information is gathered the protocol will start a
algorithm known as the Dijkstra algorithm or the Shortest Path First algorithm which is also used in GPS
navigation systems to determine the shortest route/path to any of the links/prefixes that has been
received. To do this, the router places itself as the top of the tree and then branches out to all other
links through routers to determine the fastest route.
The awareness of the entire topology for each router does impact on resources, but with the current
control-plane architectures of modern routers this is never a problem. OSPF networks can now grow up
to tens or hundreds of routers in a single area.
The behavior of OSPF changes as soon as the border of an area is reached. Information between areas is
exchanged in a way more like distance vector routing protocols do this. Information about links in the
area is summarized and then sent to the so-called backbone area (area 0). The behavior to connect
every area to the backbone area is done to prevent loops from occurring in the network. It’s now always
known that every next-hop area is area 0 of which then the decision is made where to move on next.
This means that area 0 contains all information about the OSPF network. It’s impossible to filter out any
information (except for summarization and non-OSPF external routes) from area 0. The other areas may
have limited visibility on the network. This is mostly done for a more lightweight approach to certain
areas. It means that you can connect a very small router to a highly advances OSPF backbone network
and still be able to participate in the OSPF as this lightweight router in its own area will only get very
limited information regarding the topology (usually a default route).
Information is sent through LSA messages. For the CCIE exam it’s highly recommended to know all the
different types of messages and the numbers attached to that. You have a high chance of running into
questions stating these LSA numbers, where you should know what information is carried within them
and when they are propagated in which kind of area.
7 NSSA External area Same as a type 5, but generated by an ASBR within a Not-
so-stubby area. These are only flooded within an area and
the ABR converts them to a regular Type 5.
9 Opaque 1 link Same as type 10 and 11. Used for expansion to the OSPF
protocol for carrying application information. This type is
mainly used for OSPF Graceful Restart
10 Opaque 2 area Same as type 9 and 11. Used in MPLS Fast Reroute
scenarios
During the solutions of the task more will be explained about the details regarding the different types of
areas and the different types of network links.
When looking at drawing 2. An immediate issue is spotted when looking at the areas that we need to
configure to comply with this topology. As just discussed, every area should be connected to area 0 to
prevent loops in the network.
Area 264 in this drawing is not connected to area 0 and is therefore not going to work.
There are multiple solutions to this problem, but only one of them is built into NX-OS. OSPF has a built-in
functionality to overcome these situations. In a real-life scenario this will hardly ever occur, because it’s
much easier to configure an additional interface between 2 routers and create a area 0 connection on
there. Another solution would be to use IP-IP or GRE tunnels between the routers to also ensure an
additional interface between the routers where you could run area 0 on is created.
The built-in solution we are looking for is an OSPF virtual-link. This is also some kind of tunnel
connection between 2 Area Border Routers (ABRs) to create a virtual area 0 connection between a
backbone ABR and a partitioned ABR.
Let’s start by creating the OSPF network by building up the adjacencies and advertising the Loopback
interfaces into the OSPF network.
Again the feature needs to be enabled. The Enterprise license is required to run any routing protocol,
including OSPF.
We also need to supply a tag for the OSPF process. In classical IOS this used to be a process number (ID),
but in NX-OS this has changed to a name. In the lab there is no mention of a name, so we use
IPEXPERT, but you can also use ‘1’ just like you would in IOS.
That should be sufficient to enable OSPF on the box. Now we need to enable it on the interfaces.
Verifying OSPF is enabled on all interfaces with the correct area mapping:
SW1-1(config-if)#
Now we enable the interfaces on all OSPF routers and see that OSPF adjacencies start to come up.
SW1-2
SW1-2(config-if)# feature ospf
SW1-3
SW1-3(config-if)# feature ospf
The question states that the dotted decimal notation should be used to configure area 264 in this lab.
This means that we can’t just use 265 when configuring it.
When you want to be sure to have the correct notation. You can configure it the normal way and check
a show ip ospf interface brief, as that output always gives you the dotted decimal notation
even when you configured a decimal based area number. Then you can reconfigure it again, as with
grading the proctor will see your decimal based area number instead of the dotted decimal.
Vlan134 2 0.0.1.8 40 DR 0
up
Lo0 3 0.0.0.2 1 LOOPBACK 0
up
SW1-3(config-if)#
Now the neighbor has been established, but when we check the routing table. We won’t see the
Vlan134 interface in the routing table on SW1-1. This is because the areas are not allowed to be
disconnected from area 0. The solution in this case would, as previously explained, be to create an OSPF
virtual-link to create a virtual connection to area 0 to fix this issue.
SW1-3
SW1-3(config-if)# router ospf IPEXPERT
SW1-3(config-router)# area 2 virtual-link 198.18.0.11
SW1-3(config-router-vlink)# exit
SW1-1
SW1-1(config-router)# router ospf IPEXPERT
SW1-1(config-router)# area 2 virtual-link 198.18.0.13
SW1-1(config-router-vlink)# exit
After the virtual-link is created the new VL1 adjacency automatically comes up and the network is now
seen in the OSPF routing table!
SW1-4
SW1-4(config-if)# feature ospf
SW2
SW2(config-if)# feature ospf
SW2(config-if)#
SW2(config-if)# int lo0
SW2(config-if)# ip router ospf IPEXPERT area 1
SW3
SW3(config-if)# feature ospf
After all OSPF configuration has been completed the Loopbacks are reachable throughout the network
without any problems.
The next question asks you to ignore the MTU size. This is a command to overcome MTU problems
when building up an OSPF adjacency. The OSPF adjacencies are only established when certain link
properties and configuration settings are matched on both sides.
SW1-1
SW1-1(config-if)# interface Vlan113
SW1-1(config-if)# ip ospf mtu-ignore
SW1-3
SW1-3(config-if)# interface Vlan113
SW1-3(config-if)# ip ospf mtu-ignore
The next 2 questions in the task can be solved in different ways. You will need both different ways to
solve this to get the solution correct. This is because of the behavior OSPF has by default on Ethernet
interfaces. OSPF knows various ways of determining a network connection.
Within Classical IOS the choice of different network types was huge. Mainly due to all different serial
interfaces that were available that required a different behavior (sometimes broadcast traffic wasn’t
supported for example).
Type Description
Broadcast Allows for a shared network (multi-access) with multiple routers that can
become neighbors. Uses a Designated Router for centralizing
communication.
By default all Ethernet interfaces (meaning all interfaces on an NX-OS device) are running the Broadcast
network type as it would be possible that multiple routers are sharing the same Layer 3 network
segment. This adds overhead to the set-up of the adjacency as OSPF Broadcast networks are using a
principle of a Designated Router per network segment. This Designated Router (DR) maintains
an adjacency with all routers. Other non-DR routers only have this single connection the DR router as
there OSPF adjacency. This ensures a good scaling in OSPF broadcast networks. To ensure the network is
secured to a DR failure, there is also an election of the Backup Designated Router (BDR).
Routers use a different Multicast group to communicate to the DR and BDR routers. For normal
communication the 224.0.0.5 group is used, but for DR and BDR communications the 224.0.0.6
group is used. As shown in the drawing. Routers only communicate with the DR and BDR routers. The
Hello messages from other routers on the segment are seen, but a full adjacency will not be formed. The
routers will remain in a 2WAY type of relationship with each other.
The DR election is a process which is using a Priority value to determine which router is elected. As a tie-
breaker the Router ID is used. The router with highest priority value or in case of equal priorities the
highest Router ID is elected as the Designated Router. After the DR is chosen the election is run a second
time to determine the Backup Designated Router.
To ensure that a router will never become a designated router you can change the priority value to 0.
When the priority is 0 the router will never participate in the DR or BDR election.
So we change the priority to 0 on the interfaces. Keep in mind that this change will flap your OSPF
adjacency.
SW2
SW2(config-if)# interface Vlan122
SW2(config-if)# ip ospf priority 0
SW2(config-if)# interface Vlan221
SW2(config-if)# ip ospf priority 0
SW2(config-if)# interface Ethernet1/5
SW2(config-if)# ip ospf priority 0
SW3
SW3(config-if)# interface Vlan123
SW3(config-if)# ip ospf priority 0
SW3(config-if)# interface Vlan132
SW3(config-if)# ip ospf priority 0
SW3(config-if)# interface Ethernet1/5
SW3(config-if)# ip ospf priority 0
Now the 2 uplink interfaces towards the SW1-1 and SW1-2 switches will be switching to the fact that
when you issue a show ip ospf neighbor. The neighbors will be shown as DR, meaning the local
switch is a DROTHER as the priority value ensures that SW2 and SW3 will never come a DR or BDR on
any segment.
Now the problem resides in the connection between SW2 and SW3. Both sides are now configured as
priority 0, meaning that neither of the switches will become a DR or a BDR. This now results in a non-
functional network between those 2 switches. Where we still want that adjacency to be up!
We need to solve this to get the OSPF adjacency going between SW2 and SW3.
As just mentioned, OSPF knows 2 network types in NX-OS. The Point-to-Point network type is for
Ethernet links that are directly connected between each other where no other router will ever come on
the subnet. This type of network also doesn’t know the process of electing a DR or BDR as the 2 nodes
are the only routers on the subnet.
When we configure this for the link between SW2 and SW3 we ensure that the routers establish a Full
OSPF adjacency and that none of them is a DR or BDR on that segment!
SW2
SW2(config-if)# interface Ethernet1/5
SW2(config-if)# ip ospf network point-to-point
SW3
SW3(config-if)# interface Ethernet1/5
SW3(config-if)# ip ospf network point-to-point
After configuring the point-to-point network type the adjacency is up again and everything is working
fine besides that we comply with all rules stated in the question of the task.
Next up is authentication. OSPF does support both MD5 and simple-text passwords for authenticating
the protocol. In real-life networks there is no good reason to use simple-text passwords, but for lab
purposes you should know how to use both.
Besides that OSPF also knows a difference in authenticating on a link basis or on an area basis. On area
basis the OSPF protocol requires the entire area to authenticate its adjacencies. This is a good way to
enforce authentication within an area. Link adjacencies are only dependent on the directly connected
routers so no other routers in the area know that there is authentication happening.
The first assignment indicates to secure all area 0 adjacencies. One important note here is that most
people are configuring authentication on all links between area 0 routers perfectly, but do forget a
important area 0 connection. This connection is the virtual-link crossing through area 2 to
connect area 264 to the network. This is also an area 0 adjacency and should therefore also be secured
with a hashed password! You will lose all points for this task when you forget to secure the virtual-link!
SW1-1
SW1-1(config-if)# interface Vlan112
SW1-1(config-if)# ip ospf auth
authentication authentication-key
SW1-1(config-if)# ip ospf authentication ?
<CR>
key-chain Authentication password key-chain
message-digest Use message-digest authentication
null Use null(disable) authentication
When OSPF authentication is enabled, you will enable simple-password (clear-text) password
authentication. This is not what the question stated. We should enable a hashed version of the key as
the authentication method. This is done using the message-digest authentication method. You are
also able to configure a key chain, just like with EIGRP, but it’s not mandatory and in our example we
have no reason to do so. Therefore we configure the MD5 key just on the interface. Do not configure
the OSPF authentication-key, as that would configure the simple password authentication.
Probably the authentication still works, but this is because an empty md5 hash is still an md5 hash and is
a valid key for authentication. Still you would lose all the points on this question, because the
authentication is not done using the key as stated in the question. You will need to use the message-
digest-key configuration command.
SW1-2
SW1-2(config-if)# interface Vlan112
SW1-2(config-if)# ip ospf authentication message-digest
SW1-2(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure
SW1-2(config-if)# interface Vlan211
SW1-2(config-if)# ip ospf authentication message-digest
SW1-2(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure
The final step is to enable authentication on the virtual-link as this is just an area 0 adjacency as all
other.
SW1-1
SW1-1(config-if)# router ospf IPEXPERT
SW1-1(config-router)# area 2 virtual-link 198.18.0.13
SW1-1(config-router-vlink)# ?
authentication Authentication on the vlink
authentication-key Configure the authentication key for the virtual-
link
dead-interval Dead interval
hello-interval Hello interval
message-digest-key Message digest authentication password (key)
no Negate a command or set its defaults
retransmit-interval Packet retransmission interval
transmit-delay Packet transmission delay
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1-1(config-router-vlink)# authentication ?
<CR>
key-chain Authentication password key-chain
message-digest Use message-digest authentication
null Use null(disable) authentication
We go into the sub-configuration of the virtual-link and configure the same parameters as you would on
an interface.
SW1-3
SW1-3(config-if)# router ospf IPEXPERT
SW1-3(config-router)# area 2 virtual-link 198.18.0.11
Copyright © by IPexpert. All rights reserved. 106
CCIE Data Center Lab Preparation Detailed Solution Guide
The next authentication is area based, which means that adjacencies are enforced on to run
authentication. This time it’s using simple-text passwords, so regular authentication parameters. One
difference in this configuration is that the authentication is not enabled on the interface, only the key is
specified on the interface configuration and the authentication is enabled within the routing process.
SW1-1
SW1-1(config-if)# router ospf IPEXPERT
SW1-1(config-router)# area 1 authentication
SW1-1(config-router)#
SW1-1(config-router)# interface Vlan122
SW1-1(config-if)# ip ospf authentication-key IPexpert
SW1-1(config-if)# interface Vlan221
SW1-1(config-if)# ip ospf authentication-key IPexpert
SW1-2
SW1-2(config-if)# router ospf IPEXPERT
SW1-2(config-router)# area 1 authentication
SW1-2(config-router)#
SW1-2(config-router)# interface Vlan123
SW1-2(config-if)# ip ospf authentication-key IPexpert
SW1-2(config-if)# interface Vlan132
SW1-2(config-if)# ip ospf authentication-key IPexpert
SW2
SW2(config-if)# router ospf IPEXPERT
SW2(config-router)# area 1 authentication
SW2(config-router)#
SW2(config-router)# interface Vlan122
SW2(config-if)# ip ospf authentication-key IPexpert
SW2(config-if)# interface Vlan221
SW2(config-if)# ip ospf authentication-key IPexpert
SW2(config-if)# interface Ethernet1/5
SW2(config-if)# ip ospf authentication-key IPexpert
SW3
SW2(config-if)# router ospf IPEXPERT
SW2(config-router)# area 1 authentication
SW2(config-router)#
SW2(config-router)# interface Vlan123
SW2(config-if)# ip ospf authentication-key IPexpert
SW2(config-if)# interface Vlan132
SW2(config-if)# ip ospf authentication-key IPexpert
SW2(config-if)# interface Ethernet1/5
SW2(config-if)# ip ospf authentication-key IPexpert
The next question is asking about summarization. Now summarization is different in OSPF compared to
EIGRP. Distance vector protocols send their routing table towards other routers in the network. A link-
state protocol will be default only advertise its link information. Every router needs to have a full view
on the network before it can start calculating the shortest path to all destinations in the network.
Now as earlier discussed the OSPF routing protocol uses a distance vector style way of advertising
networks to other areas. This means that we can summarize at the borders of areas, before injecting
routes into area 0.
This is done using the area-range command in the OSPF protocol configuration. We first need to
generate additional Loopback interfaces on SW2.
SW2
SW2(config)# int lo1
SW2(config-if)# ip add 198.18.128.1/24
SW2(config-if)# no shut
SW2(config-if)# int lo2
SW2(config-if)# ip add 198.18.129.1/24
SW2(config-if)# no shut
SW2(config-if)# int lo3
SW2(config-if)# ip add 198.18.130.1/24
SW2(config-if)# no shut
SW2(config-if)# int lo4
SW2(config-if)# ip add 198.18.131.1/24
SW2(config-if)# no shut
• 198.18.128.0/24
• 198.18.129.0/24
• 198.18.130.0/24
• 198.18.131.0/24
We see that the third octet is changed. In binary the third octet looks like this:
• 10000000
• 10000001
• 10000010
• 10000011
Anything between the last 2 bits is changed in these static routes. So when we create a static route with
a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!
We need to get the networks into OSPF. This can be done in 2 ways. One by configuring the Loopback
interfaces to appear in the OSPF database and the other is by doing redistribution
Be aware that the summary configuration for both ways is very different! As for the first way the area-
range command is used and for the second the summary-address command. In the question it’s
stated that the single entry should be seen as a Type 3 LSA in the backbone. When an External route is
summarized the summary is also advertised as a Type 5. It’s therefore necessary to use the area-range
style configuration as that will inject a regular Type 3 LSA instead of a Type 5.
SW2
SW2(config)# int lo1
SW2(config-if)# ip ospf network IPEXPERT area 1
SW2(config-if)# int lo2
SW2(config-if)# ip ospf network IPEXPERT area 1
SW2(config-if)# int lo3
SW2(config-if)# ip ospf network IPEXPERT area 1
SW2(config-if)# int lo4
SW2(config-if)# ip ospf network IPEXPERT area 1
Now SW2 is not the border router for the network. Summarization can only be done on the area border
routers. Therefore the area-range command is not effective when configured on SW2, but only on SW1-
1 and on SW1-2. Don’t forget you have 2 ABR’s in area 1!
SW1-1
SW1-1(config-if)# router ospf IPEXPERT
SW1-1(config-router)# area 1 range 198.18.128.0/22
SW1-2
SW1-2(config-if)# router ospf IPEXPERT
SW1-2(config-router)# area 1 range 198.18.128.0/22
After enabling the area range. Other areas will now only see a single Type 3 LSA for these 4 networks
instead of the specific ranges!
The next question states to ensure the whole subnet is seen in the routing table. Isn’t this always done?
Well in classical IOS this was the case, but in NX-OS the behavior of Loopback interfaces has changed.
There is not a real reason why you would want to put an other than /32 mask on a Loopback interface,
but it is possible in all Cisco platforms.
SW1-3
SW1-3(config-if)# interface Loopback1
SW1-3(config-if)# ip address 198.18.13.1/24
SW1-3(config-if)# ip router ospf IPEXPERT area 2
SW1-3(config-if)# no shutdown
Copyright © by IPexpert. All rights reserved. 109
CCIE Data Center Lab Preparation Detailed Solution Guide
By default all Loopback interfaces will be advertised with a /32 mask, no matter what mask has been
configured on the Loopback interface itself.
SW1-3
SW1-3(config-if)# interface Loopback1
SW1-3(config-if)# ip ospf advertise-subnet
After configuring this command the route is now shown with its correct /24 mask in the routing table on
other routers.
There are several types of stub areas. Each with a different behavior. Below is a table explaining the
different types of areas and the behaviors that are related to it.
Type Description
Stub Does not allow external routes (Type-5 LSA’s) in the area. Default
route ‘can’ be advertised by the ABR to ensure full reach ability.
Totally Stub Same as a stub area, so not allowing external routes (Type 5 LSA’s).
Besides that Summary routes (Type-3 LSA’s) are not allowed as well.
Meaning that there is only reach ability within the area and the ABR’s
advertise a default route into the area to ensure the routers in that
area can still reach the rest of the network. This default is a Type-3
LSA. Meaning 1 Type-3 LSA is allowed in a Total Stub area.
Not-so-stubby Same as a stub area, so not allowing external routes (Type 5 LSA’s).
But do allowing ASBR’s in the area. The result is that an ASBR within a
not-so-stubby area (NSSA) will inject Type 7 LSA’s in the NSSA. These
will be converted to a regular Type 5 on the ABRs of that area.
Totally Not-so-stubby Same as a not-so-stubby area, so not allowing Type 5 LSA’s to exist in
the area, but allowing ASBR’s. Then Type-3 LSA’s are also not allowed
as in a Totally Stub area. This means a default route is injected in the
area to ensure reach ability to the rest of the network.
The drawing below illustrates the above table. Where ASBR’s can be placed in the network. Remind that
area 0 can never be altered to a stub or not-so-stub area! What is not explained in the drawing, but is
possible is that ABR’s are ASBR’s at the same time.
Now back to the question. The question states the following characteristics of area 1:
• No Type 5 LSA
• No Type 4 LSA
o This is also included in the Stub area. As there are no Type 5 LSA’s in the area, it’s also
not required to have a Type 4 LSA (reach ability to the ASBR) in the area
• No Type 3 LSA
o This points to a Totally Stub area, where all Type 3 LSA’s are filtered from crossing the
ABRs and a Default route is advertised in to the area. This would solve the question.
Except that the Default route that is injected into a Totally Stub area is still a Type 3 LSA.
The question clearly states that this is not allowed
We can solve this behavior. Within a Totally NSSA area we do not allow Type 3,4 and 5 LSA’s. So it’s
equal to a Totally Stub area. The difference is that we can introduce a new type of LSA in the area which
Copyright © by IPexpert. All rights reserved. 112
CCIE Data Center Lab Preparation Detailed Solution Guide
is a Type 7 for the External routes that are injected when the NSSA area contains an ASBR. These LSA’s
are interpreted by every router in the area and therefore they install those external routes from the
ASBR in the NSSA area.
What we can do at the ABRs is to ensure that the Default route they inject into the Totally NSSA area is
not a Type 3 LSA (as it would be by default on IOS). It can be a Type 7 LSA. Be aware that you need to
configure this and this is not supported on every IOS release as well. On NX-OS however it’s the default!
The configuration for this is relatively easy and is easy to spot with use of the question mark.
SW1-1
SW1-1(config)# router ospf IPEXPERT
SW1-1(config-router)# area 1 ?
authentication Enable authentication for the area
default-cost Specify default-cost for default summary LSA
filter-list Filter prefixes between OSPF areas
nssa Configure area as NSSA
range Configure an address range for an area
stub Configure area as a stub
virtual-link Define a virtual link and its parameters
On the other routers within the area we do not have to configure any of the default route information.
What’s important for the adjacency to come up is that the area is configured as an NSSA area. This is
advertised within the adjacency packets. Therefore we only require the following information on the
other switches.
SW2
SW2(config)# router ospf IPEXPERT
SW2(config-router)# area 1 nssa
SW3
SW3(config)# router ospf IPEXPERT
SW3(config-router)# area 1 nssa
Before applying the NSSA configuration the routing table shows more specific routes within area 1.
After applying the configuration every route is summarized into the default route.
When checking the OSPF database, there are no Type 3, 4 or 5 LSA’s visible so we accomplished this
question!
The final question of this task states to ensure that all OSPF routers do not attract traffic just after start-
up of the devices. This should last for 2 minutes. The command we are looking for is the max-metric
command, where you can add several values of how this max metric is configured.
The result is that once the router boots up and OSPF adjacencies come online. The router will advertise
everything with a maximum metric value until the timer expires. Default this timer is 600 seconds, but is
now adapted to 120.
SW1-1
SW1-1(config-if)# router ospf IPEXPERT
SW1-1(config-router)# max-metric router-lsa on-startup 120
New command will take effect at next reload
SW1-2
SW1-2(config-if)# router ospf IPEXPERT
SW1-2(config-router)# max-metric router-lsa on-startup 120
SW1-3
SW1-3(config-if)# router ospf IPEXPERT
SW1-3(config-router)# max-metric router-lsa on-startup 120
SW1-4
SW1-4(config-if)# router ospf IPEXPERT
SW1-4(config-router)# max-metric router-lsa on-startup 120
SW2
SW2(config-if)# router ospf IPEXPERT
SW2(config-router)# max-metric router-lsa on-startup 120
SW3
SW3(config-if)# router ospf IPEXPERT
SW3(config-router)# max-metric router-lsa on-startup 120
Pay close attention when using redistribution in your topology. You can easily run into issues and cause
routing loops in the network. This is especially the case for protocols without distinction in route
preference / administrative difference like OSPF. The active route is the one with the lowest
Administrative Distance. The following table describes the default values which can be changed per
protocol. The change can be done for certain routes, certain upstream routers or types of routes
(internal/external/local).
Static 1
EIGRP Summary 5
EIGRP 90
OSPF 110
IS-IS 115
RIP 120
Unreachable 255
Some protocols distinct routes that are internal to their own protocol and prefixes that were learnt via
redistribution. OSPF does not do this by default, but can be configured to ensure Type 5 LSA prefixes will
have another Administrative Distance attached. In most cases this is even recommended. It’s easy to get
a routing loop when there is no distinction between the administrative distances for internal and
external routes.
Copyright © by IPexpert. All rights reserved. 116
CCIE Data Center Lab Preparation Detailed Solution Guide
In our network we will also run into some issues and these will cause issues (no connectivity issues).
With a normal redistribution the EIGRP learned routes will be learned back by the other router back into
EIGRP. The mechanism that EIGRP uses in this case is that Externally learned routes will have a higher
administrative distance attached to the routes and therefore select their primary internal routes as the
ones where traffic will be send to.
Within the OSPF network there will be issues with routes learning 2 destinations.
We first configure the switches for redistribution on 2 positions in the network. We are required to
configure a route-map for the redistribution to ensure only routes are advertised that you really want
to.
The drawing below illustrates what is about to happen when we do not enter some filtering information.
To ensure that routes are not redistributed back their to their own protocol.
Now we configure the switches with filtering and tagging in place so that routes that are advertised to
either EIGRP or OSPF are tagged with a value equal to the Administrative Distance (for easier
recognition, but this could be any value). On the next redistribution point this value gets rejected and
others get tagged. This way we have a very easy way to filter routes and ensure that we do not cause
loops in our redistribution scenario.
SW1-2
SW1-2(config)# route-map E->O deny 10
SW1-2(config-route-map)# match tag 110
SW1-2(config-route-map)# route-map E->O permit 20
SW1-2(config-route-map)# set tag 90
SW1-2(config-route-map)# exit
SW1-2(config)# router ospf IPEXPERT
SW1-2(config-router)# redistribute eigrp IPEXPERT route-map E->O
SW1-2(config-router #
SW1-2(config-if)# route-map O->E deny 10
SW1-2(config-route-map)# match tag 90
SW1-2(config-route-map)# route-map O->E permit 20
SW1-2(config-route-map)# set tag 110
SW1-2(config-route-map)# exit
SW1-2(config)# router eigrp IPEXPERT
SW1-2(config-router)# redistribute ospf IPEXPERT route-map O->E
SW1-4
SW1-4(config)# route-map E->O deny 10
SW1-4(config-route-map)# match tag 110
SW1-4(config-route-map)# route-map E->O permit 20
SW1-4(config-route-map)# set tag 90
SW1-4(config-route-map)# exit
SW1-4(config)# router ospf IPEXPERT
SW1-4(config-router)# redistribute eigrp IPEXPERT route-map E->O
SW1-4(config-router #
SW1-4(config-if)# route-map O->E deny 10
SW1-4(config-route-map)# match tag 90
SW1-4(config-route-map)# route-map O->E permit 20
SW1-4(config-route-map)# set tag 110
SW1-4(config-route-map)# exit
SW1-4(config)# router eigrp IPEXPERT
SW1-4(config-router)# redistribute ospf IPEXPERT route-map O->E
Routes being redistributed between protocols and re-redistributing routes back into their original
protocol. Due to the intelligence in most protocols this will not cause any harm and traffic will be
forwarded. Still it will cause a lot of ‘dirt’ in the routing protocol databases.
Now we have successful redistribution and have a full reach ability throughout the network without any
loops!
The next couple tasks are regarding the Bidirectional Forwarding Detection (BFD) protocol. This protocol
ensures a very fast detection that a neighbor router is down, regardless of link status.
Usually the link status is used for detecting a neighbor down event. This is not reliable in situations
where for example a Ethernet circuit or DWDM circuit, without propagation of link-loss to client ports,
the routers will not know if something happens within the circuit.
The BFD protocol is the Cisco best practice for a fast detection mechanism. The protocol itself is just a
very basic Layer 3 protocol which sends very fast Hello messages. Once a certain threshold (by default 3)
is reached a trigger is given. The BFD protocol is independent of the routing protocol and can actually be
used for any protocol when supported (for example HSRP). Multiple protocols can be attached to the
same BFD session, making it a very lightweight protocol for detecting these neighbor loss events as only
a single session is running even when multiple protocols are used across the link between the 2 routers.
The above illustration shows the BFD session being totally independent of the OSPD adjacency running
on the router. Once the BFD session is detected to be offline, the OSPF protocol is signaled that this
specific router has gone offline and therefore OSPF cuts of the adjacency right away, before waiting on
its own Holddown-timer thresholds. Multiple protocols can be signaled via this single BFD session.
One additional advantage of the BFD protocol is that it’s so simple, that all Hello packets are generated
and handled within the Hardware ASICs. The Nexus 7000 and Nexus 5500 interfaces are all able to
handle BFD within the hardware and therefore BFD has no impact on the CPU load of the control-plane.
Sub-second hello timers in OSPF due have impact on the CPU load!
As the BFD protocol is just an IP protocol, it can also be secured, just like the mainstream dynamic
routing protocols with an MD5 hash. Detection is by default using a hello timer of 300ms and a
threshold of 3 missed hello packets, before declaring a neighbor down.
In this case we only need to ensure that failure detection of SW1-2 happens very fast. We will use the
BFD protocol for this. We first need to configure the BFD session under the interface and after that we
configure the routing protocol to converge along with the BFD protocol. As the BFD protocol is also a
Layer 3 protocol, we need to enable it on the NX-OS device first.
SW1-2
SW1-2(config)# feature bfd
Please disable the ICMP redirects on all interfaces
running BFD sessions using the command below
We need to configure no ip redirects as a best practice. This has to do that it might be possible
for an interface to return BFD hellos with a redirect (when not properly configured). This might cause
that a lot of packets are send and need to be answered with an ICMP message. All ICMP traffic is
handled by the control-plane and not in the ASIC as BFD messages are. Therefore it’s highly
recommended to disable this.
There are a number of things that need to be configured when configuring BFD sessions. You
immediately require configuring the timers of the session. When you look a little further to the
questions in this task it’s stated that the sessions should detect failures within 300ms. Therefore we are
configuring hello-intervals of 100ms and a multiplier of 3. Meaning the device can miss 3 hello’s before
declaring the neighbor down. With hellos of 100ms, this occurs at a maximum of 300ms, which after the
connected protocols are signaled.
What remains is that the OSPF and EIGRP protocols need to be configured for the BFD protocol and we
need to enable the interfaces that we want to use the protocol on.
If we would be configuring BGP, we would configure these options on the Neighbor sessions. The BFD
session configuration would still be done under the interface. Remember to configure the correct
routing protocols and also the correct interface configuration!
SW1-2
SW1-2(config)# router ospf IPEXPERT
SW1-2(config-router)# bfd
SW1-2(config-router)# router eigrp IPEXPERT
SW1-2(config-router)# bfd
SW1-2(config-router)# interface Vlan132
SW1-2(config-if)# ip ospf bfd
SW1-2(config-if)# interface Vlan123
SW1-2(config-if)# ip ospf bfd
SW1-2(config-if)# interface Vlan112
SW1-2(config-if)# ip ospf bfd
SW1-2(config-if)# interface Vlan211
SW1-2(config-if)# ip ospf bfd
SW1-2(config-if)# interface Vlan124
SW1-2(config-if)# ip eigrp IPEXPERT bfd
Now we configured only 1 router. BFD consists of that both sides need to be configured before
everything works. We now configure the other sides of the interfaces that connect to SW1-2.
SW1-1
SW1-1(config)# feature bfd
SW1-1(config)# router ospf IPEXPERT
SW1-1(config-router)# bfd
SW1-1(config-router)# exit
SW1-1(config-if)# interface Vlan112
SW1-1(config-if)# no ip redirects
SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW1-1(config-if)# ip ospf bfd
SW1-1(config-if)# interface Vlan211
SW1-1(config-if)# no ip redirects
SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW1-1(config-if)# ip ospf bfd
SW3
SW3(config)# feature bfd
SW3(config)# router ospf IPEXPERT
SW3(config-router)# bfd
SW3(config-router)# exit
SW3(config-if)# interface Vlan123
SW3(config-if)# no ip redirects
SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW3(config-if)# ip ospf bfd
SW3(config-if)# interface Vlan132
SW3(config-if)# no ip redirects
SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW3(config-if)# ip ospf bfd
SW1-4
SW1-4(config-router)# router eigrp IPEXPERT
SW1-4(config-router)# bfd
SW1-4(config-router)# exit
SW1-4(config)# interface Vlan124
SW1-4(config-if)# ip eigrp IPEXPERT bfd
The questions also state that the BFD session between SW1-2 and SW3 should be secured using an SHA-
1 hash. We can configure this under the interfaces. Don’t forget that there are 2 interfaces and
therefore 2 sessions between the switches, which both need to be secured.
SW1-2
SW1-2(config)# interface Vlan123
SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW1-2(config)# interface Vlan132
SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW3
SW3(config)# interface Vlan123
SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW3(config)# interface Vlan132
SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
The first task is to configure a static ARP mapping on a specific interface. In Classical IOS this
configuration was done in Global Configuration mode. Within NX-OS this is done under the interface
where the ARP entry is reachable. This means that it’s not necessary to include an interface anymore in
the ARP command, but you only need to configure which mac addresses are found on which interface.
SW1-1
SW1-1(config-if)# int vlan 112
SW1-1(config-if)# ip arp ?
A.B.C.D IP address
gratuitous Gratuitous
timeout ARP timeout
version 5.1(5)
interface Vlan112
no shutdown
ip address 198.18.112.11/24
ip arp 198.18.112.24 ABCD.1234.5678
SW1-1(config-if)#
We configured the static ARP entry directly under the interface and completed the question.
Next is a bit cryptic description regarding Gratuitous ARPs. With this mechanism the switch will
broadcast an ARP entry for its own IP address and checks if anybody replies with a different MAC
address (different than its own local interface). After which it can send a warning about a duplicate IP
address on the segment.
This mechanism is enabled by default. Though one configuration option, to actively update its cache is
disabled. This is what the question asks for.
SW2
SW2(config-if)# int Ethernet1/5
SW2(config-if)# ip arp gratuitous ?
request Enable/Disable sending grat. arp request when duplicate address
detected
update Enable/Disable arp cache updates for gratuitous arp
The next question is also a bit cryptic which means you have to read the Unicast Routing configuration
guide pretty well to find this. The Nexus 7000 keeps a memory reservation for outstanding ARP entries
in its Forwarding Information Base (FIB). This memory is cleared after a while. This is done to protect the
memory from overflowing with ARP entries when an attack is started (flooding of MAC addresses by an
attacker to overflow the memory of switches).
This is called ARP throttling or Glean throttling (Glean ARPs are incomplete ARP entries
waiting for a reply) and is configured using special hardware commands.
SW1-1
SW1-1(config)# hardware ?
access-list Access Control List
ejector Card ejector functionality
fabrics Fabrics supported in the switch
forwarding Forwarding information
ip IP
ipv6 (no abbrev) IPv6
proxy Proxy Forwarding Information
rate-limiter Configure Rate-Limiter for packets forwarded to
supervisor
SW1-1(config)# hardware ip ?
glean Glean
verify Enable IPv4 and some IPv6 packet validation checks in hardware
Remind to first enable the feature and then change the maximum.
The final question is regarding RFC 1191 to be enabled on all switches. A quick search through the
documentation (Ctrl-F or Cmd-F) learns that this is related to TCP Path-MTU-Discovery! Easily
enabled on all switches with a single command.
SW1-1
SW1-1(config)# ip tcp path-mtu-discovery
SW1-2
SW1-2(config)# ip tcp path-mtu-discovery
SW1-3
SW1-3(config)# ip tcp path-mtu-discovery
SW1-4
SW1-4(config)# ip tcp path-mtu-discovery
SW2
SW2(config)# ip tcp path-mtu-discovery
SW3
SW3(config)# ip tcp path-mtu-discovery
Task 6 is completed! Now before moving on with the next Task it’s recommended to take a little break
as the next task requires loading a new initial configuration on the switches. This will take some time,
approximately 10-15 minutes, as the switches need to be reloaded to load a new VDC configuration with
other mappings and pre-configuration.
We purposely excluded newer generation linecards (F2 and M2) as these are not in the hardware
blueprint of the CCIE Data Center and therefore not in scope of this workbook.
The questions are very basic formulated in this question. This is due to the fact that the technology
underneath is quite complex and very groundbreaking to begin with. Still Cisco made a great effort in
making this one of the easiest things to configure on the box.
FabricPath and OTV are literally just a few commands to enable and get it up and running, just the way
you want. Modifications can be done to the protocols, but these are limited. Also in scope of the CCIE
Data Center, the amount of topologies that can be created with these technologies is limited.
Before you know what exactly to configure to stretch the layer 2 VLAN across multiple datacenters over
a Fabric Path and OTV network, you need to know some structure of the protocols.
FabricPath
The FabricPath protocol is a Cisco proprietary protocol for making Layer 2 routed instead of switched.
This is done by eliminating ARP from the network and implementing IS-IS as routing protocol to
advertise reach ability in the network. This way there is no broadcast traffic occurring when it’s not
necessary. FabricPath certainly supports all various ways of Multicast and Broadcast traffic, but it’s sent
in a very optimal way.
The next big step is that it’s not necessary for all switches to have knowledge about all MAC addresses in
the network. Therefore a mechanism is introduced called Conversational MAC learning. This
mechanism is only available on the F-linecards of the Nexus 7000 switches and on the Nexus 5500 series
switches. It means that switches only remember the MAC address information, once there is a
bidirectional traffic flow. Instead of just learning the MAC address once a packet is seen from it.
The IS-IS routing protocol is totally invisible for the FabricPath protocol and for the CCIE configuring it.
Only when troubleshooting is necessary, certain commands related to IS-IS are required to check if
adjacencies are up and that topology information is exchanged.
The FabricPath protocol works with an additional encapsulation on the packet (the protocol kind of
looks like MPLS). The encapsulation is a ‘FabricPath’ encapsulation. Which means that it’s an
adjusted Ethernet frame that is added in front of the packet. FabricPath is also called a “MAC-in-MAC”
protocol. This header is modified for FabricPath and therefore it’s also NOT supported to run FabricPath
over an Ethernet circuit or to have Switches in between the path. The connection between 2 FabricPath
devices needs to be a direct (dark) fiber connection to support it.
There is an IEEE initiative called TRILL that is very much alike FabricPath in many ways. The main
difference is that TRILL does use a standard Ethernet header as the encapsulation and therefore does
support that switches or Ethernet circuits are in between the TRILL devices. This enables a lot more use
of the protocol and it’s not necessary to have all TRILL enabled devices in your network. For example
you could upgrade your access or distribution switches to TRILL and have your core still run as standard
Ethernet switches. They switch the TRILL traffic and do not have any awareness of what’s inside the
packet and what kind of high advanced technology is happening on the Edge of the network.
All this sounds very familiar when you are used to work with MPLS networks. Where also the core is a
very ‘dumb’ device that just forwards packets to the right interfaces and that all intelligence of the
network is inside the network edge. This also accounts for both FabricPath and TRILL.
Below is a drawing explaining the traffic flow in-case of a known-unicast, meaning that ARPs have
successfully passed along. In case of an unknown unicast, the ARP request of the access device or end-
host is flooded throughout the network. This is done in an optimal way using so-called Broadcast Trees.
Every edge switch (called leaf switches in FabricPath) uses a dedicated tree of switches, calculated using
the SPF algorithm of IS-IS where it will forward the traffic along. By using the Broadcast tree, every
switch in the network receives the packet only once and never multiple times, as could be the case with
ARPs and normal broadcasts.
The destination host will reply to the ARP and send it back along the same distribution / broadcast tree
through the FabricPath network. Once traffic starts being sent, the network learns about this flow and
will remember the traffic flow in its tables (conversational learning). In that case the first leaf switch
knows about where the packet should go and sends it unicast to the next switch. Where it’s
decapsulated and another lookup is done where the destination MAC address is located on that
particular switch. The packet is then forwarded out in plain Ethernet to the end-host.
The drawing explains this traffic flow in a very easy way so the flow is known to you.
2. Leaf 1 receives the packet and sees that it’s destination (PC 2) is on Leaf 2
3. Leaf 1 encapsulates the packet with a FabricPath header, with a destination address of Leaf
2 and a source address of Leaf 1
4. The FabricPath network forwards the packet based on the FabricPath header to Leaf 2
without looking at the rest of the packet headers
5. Leaf 2 decapsulates the packet and does a lookup on the Destination of the Ethernet frame
inside of the FabricPath frame
By using this method of traffic forwarding the core network has no awareness of what’s happening
inside of the FabricPath frame and can therefore have a very limited FIB table. The core network only
needs to know the way to the destination leaf switches, which is advertised using a modified version of
IS-IS.
Now on to the configuration. We also configure some SVI’s to test the configuration and ensure hosts on
VLAN 666 can reach the rest of the network by using FabricPath. As you can see in the network drawing,
there are no direct connections between the switches with a Classical Ethernet encapsulation. So the
FabricPath network needs to work before replies are received from other hosts on the network.
SW2
SW2(config)# vlan 666
SW2(config-vlan)# exit
SW2(config)# feature interface-vlan
SW2(config)# interface vlan666
SW2(config-if)# ip address 198.18.66.1/24
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW3
SW3(config)# vlan 666
SW3(config-vlan)# exit
SW3(config)# feature interface-vlan
SW3(config)# interface vlan666
SW3(config-if)# ip address 198.18.66.2/24
SW3(config-if)# no shutdown
SW3(config-if)# exit
SW1-3
SW1-3(config)# vlan 666
SW1-3(config-vlan)# exit
SW1-3(config)# feature interface-vlan
SW1-3(config)# interface vlan666
SW1-3(config-if)# ip address 198.18.66.3/24
SW1-3(config-if)# no shutdown
SW1-3(config-if)# exit
SW1-4
SW1-4(config)# vlan 666
SW1-4(config-vlan)# exit
SW1-4(config)# feature interface-vlan
SW1-4(config)# interface vlan666
SW1-4(config-if)# ip address 198.18.66.3/24
SW1-4(config-if)# no shutdown
SW1-4(config-if)# exit
FabricPath only works on the F1 linecards. The encapsulation and FabricPath VLANs can only live on the
F1 linecard and can never be configured on an M1 linecard. However, the routing feature of the M1
linecard can be ‘borrowed’ by the F1 linecards. Traffic entering on the FabricPath interface will be
directed over the Fabric towards the M1 linecard to ensure that layer 3 traffic is properly handled by the
switch.
On the Nexus 5500 series, a similar configuration is done by using the Layer 3 daughter module
that is also inserted like an interface in the router. Therefore the layer 2 only ASICs can forward traffic to
the layer 3 module and comes back again to be re-encapsulated into the FabricPath network.
SW2
SW2(config)#
SW2(config)# feature-set fabricpath
SW2(config)# vlan 666
SW2(config-vlan)# mode fabricpath
SW3
SW3(config)#
SW3(config)# feature-set fabricpath
SW3(config)# vlan 666
SW3(config-vlan)# mode fabricpath
SW1-3
SW1-3(config)#
SW1-3(config)# feature-set fabricpath
SW1-3(config)# vlan 666
SW1-3(config-vlan)# mode fabricpath
SW1-4
SW1-4(config)#
SW1-4(config)# feature-set fabricpath
SW1-4(config)# vlan 666
SW1-4(config-vlan)# mode fabricpath
SW1-1
SW1-1(config)#
SW1-1(config)# feature-set fabricpath
SW1-2
SW1-2(config)#
SW1-2(config)# feature-set fabricpath
With enabling the fabricpath feature-set, you enable a whole suit of protocols in a single command. For
example IS-IS is started and the FP protocol itself. What’s also started is an automatic switch ID
selection. It’s possible to manually configure the switch ID’s (where traffic is forwarded to in the FP
frames), but the automatic selection will pick values which are unique in the network. Therefore we will
leave it as automatic in our network.
We also configure the VLAN which is to be extended through the FabricPath network (VLAN 666) as a
FabricPath VLAN. This is to configure the VLAN that it’s allowed to travel the FabricPath network.
By default all VLANs remain in so-called ‘CE’ (Classical Ethernet) mode, so will never cross the FabricPath
network.
Now what’s left is the configuration of the interfaces that are pointing to the other FabricPath switches.
Ensure that you do not configure any other interfaces with this encapsulation as a lot of processes will
be configured under this interface and for example IS-IS will start attempting to make adjacencies on
these interfaces.
SW2
SW2(config)#
SW2(config)# interface Ethernet1/1
SW2(config-if)# switchport mode fabricpath
SW2(config)# interface Ethernet1/5
SW2(config-if)# switchport mode fabricpath
SW3
SW3(config)#
SW3(config)# interface Ethernet1/1
SW3(config-if)# switchport mode fabricpath
SW3(config)# interface Ethernet1/5
SW3(config-if)# switchport mode fabricpath
SW1-3
SW1-3(config)#
SW1-3(config)# interface Ethernet2/9
SW1-3(config-if)# switchport mode fabricpath
SW1-3(config)# interface Ethernet2/11
SW1-3(config-if)# switchport mode fabricpath
SW1-4
SW1-4(config)#
SW1-4(config)# interface Ethernet2/13
SW1-4(config-if)# switchport mode fabricpath
SW1-4(config)# interface Ethernet2/15
SW1-4(config-if)# switchport mode fabricpath
SW1-1
SW1-1(config)#
SW1-1(config)# interface Ethernet2/1
SW1-1(config-if)# switchport mode fabricpath
SW1-1(config)# interface Ethernet2/3
SW1-1(config-if)# switchport mode fabricpath
SW1-2
SW1-2(config)#
SW1-2(config)# interface Ethernet2/5
SW1-2(config-if)# switchport mode fabricpath
SW1-2(config)# interface Ethernet2/7
SW1-2(config-if)# switchport mode fabricpath
After a while our interface should come up, IS-IS adjacencies have formed and our FabricPath network is
up and running!
The next and final step of this chapter is to configure the inter-datacenter communication. This is
accomplished using another Nexus 7000 specific Cisco Proprietary protocol called Overlay
Transport Virtualization.
The OTV protocol works, from a data plane perspective, very similar as FabricPath works. The next-hop
of the MAC address-table is changed from a destination interface to a Destination IP address instead of
a Destination MAC address as it would be in a FabricPath network.
The OTV protocol uses IP encapsulation instead of Ethernet. Different from FabricPath is that OTV is
created to run over standard IP networks. This means that the only requirement you have between data
centers to extend layer 2 between them is a Layer 3 connection with IPv4 routing. Any IP connection
between the data centers is enough to ensure that OTV can function.
While FabricPath is called a MAC-in-MAC protocol. OTV can be called a MAC-in-IP protocol.
OTV is designed to support numerous data centers that share the same Overlay network. Meaning that
one VLAN can be extended across many different data centers. This means that the OTV network is
optimized to work in a Multicast enabled environment. The configuration becomes quite complex
when unicast is used. In our case with only 2 devices and a directly connected link we will certainly use
Multicast for our solution.
The unicast model works with a central node (or 2) that will be in charge of maintaining adjacencies with
all sites for spreading all control-plane information throughout the Overlay network. The model looks a
bit like the OSPF DR and BDR principle or the BGP route reflector principle, where devices
only maintain 2 connections to these devices and with nobody else. These primary devices will exchange
the information about the network to all nodes.
As previously mentioned the protocol works very similar to FabricPath in traffic flow in the data plane,
so we will not describe this in further detail. Only the final traffic flow will be shown in a diagram.
OTV requires a little more configuration than FabricPath, as it has no knowledge of the rest of the
network and needs some IP addressing to be configured. Besides that it requires an Overlay network to
be configured.
OTV also uses the IS-IS routing protocol to exchange the MAC address information between all nodes.
This information is exchanged through Multicast messages. This IS-IS protocol is a different instance
than the IS-IS protocol used for FabricPath. This is done to ensure they will never interfere with each
other.
4. A block of SSM (Source Specific Multicast) groups (8) needs to be selected for transferring
multicast and broadcast traffic of the network across the Overlay.
6. A local VLAN (site VLAN) needs to be selected for local control-plane traffic within the
datacenter. Mainly used for split-brain detection and loop prevention. The local VLAN may never
be extended to other sites!
7. A site-identifier to ensure the site is uniquely numbered. When using multiple OTV
nodes at a single site, this number should match on all devices on that site.
OTV only works on the M1 linecards of the Nexus 7000 series switches. This line card is typically used in
core switch scenarios where also the layer 2 to layer 3 border is placed. Meaning that SVI’s are
configured on the core switches to terminate layer 3 routing of the datacenter network.
There is one important limitation with OTV! The switch that is configured for OTV may not contain any
SVI’s of the VLANs that are extended to other sites! This is a hardware limitation in the M1 line cards. In
future software and/or hardware releases of NX-OS and/or linecards it’s expected that this limitation is
removed. The current workaround is to use a different VDC on the Nexus 7000 just for OTV extension. In
our network we do not have an SVI of the extended VLAN configured on that switch (VDC) therefore we
do not run into this limitation.
First we configure the required configuration that will support the OTV protocol.
SW1-1
SW1-1(config)#
SW1-1(config)# interface Ethernet1/10
SW1-1(config-if)# no switchport
SW1-1(config-if)# ip address 198.18.10.1/24
SW1-1(config-if)# ip igmp version 3
SW1-1(config-if)# no shutdown
SW1-1(config-if)#
SW1-1(config-if)# vlan 666
SW1-1(config-vlan)# mode fabricpath
SW1-2
SW1-2(config)#
SW1-2(config)# interface Ethernet1/17
SW1-2(config-if)# no switchport
SW1-2(config-if)# ip address 198.18.10.2/24
SW1-2(config-if)# ip igmp version 3
SW1-2(config-if)# no shutdown
SW1-1(config-if)#
SW1-1(config-if)# vlan 666
SW1-1(config-vlan)# mode fabricpath
We are told to use the 198.18.10.0/24 subnet for a Layer 3 connection. This will be the only layer 3
connection in our topology, so we can use the entire /24 network if we want. We also enable IGMP to
ensure Multicast processing is enabled on the interface. OTV requires version 3 of IGMP to work
correctly. At this stage we also enable the VLAN that is about to be extended to the other site.
Meanwhile it’s also the FabricPath VLAN as the traffic coming in for this VLAN is coming in via a
FabricPath enabled network in our topology.
SW1-1
SW1-1(config)#
SW1-1(config)# feature otv
We are required to configure a site-identifier for OTV. In FabricPath the switch ID’s are
automatically selected, but for OTV the site-ID needs to be equal on all nodes within the same data
center (site), therefore this needs to be configured manually.
The site-vlan is used for sending local control-plane traffic between the OTV enabled switches. This
VLAN may never be extended across the Overlay to other nodes in the OTV enabled network. The
information sent in this VLAN is used for loop prevention and split-brain detection. By default VLAN 1 is
used for this. It’s highly recommended to change this to a dedicated VLAN. There were no indications in
the question which value is required to be used, so we can pick any number. We also don’t have any
other OTV enabled switches in this network, so the VLAN is unused in this topology, however required in
the configuration.
The Overlay interface is the interface used for the encapsulation onto the OTV enabled network.
All OTV related configuration is done under this interface.
The control-group is the multicast group that is used for control-plane traffic sent across the OTV
network to exchange all information to other OTV nodes. This can be any multicast group that is
forwarded correctly through the multicast enabled network.
The data-group is the range of multicast groups, with a maximum of 8, on which the multicast and
broadcast traffic of the extended VLANs is sent. These groups should be from the Source Specific
Multicast (SSM) range. Again we didn’t have any value given, so we can use any /28 subnet to ensure we
have 8 groups available for multicast/broadcast transmission across the OTV network.
The last option in this snippet is the join-interface. This is the IP interface (interface configured
with an IP address) that is used for sending the IP encapsulated traffic on. You can only configure 1
Ethernet interface with this command. When resiliency and/or load-balancing is required there are 2
options. The first is to use a Port-channel of M1 linecard interfaces as the join-interface. The second
is to use 2 or more Overlay interfaces with different OTV parameters, but with the same VLAN
extension configured.
The final step in the configuration is to configure the VLANs that are going to be extended to the
other data center. In our topology this is VLAN 666 which is extended. All the above mentioned
parameters on the Overlay1 interface need to be configured first, before the no shutdown
command is issued. When some pieces of the configuration are missing, the system will report an error,
or the OTV network will not work properly.
Once the OTV network is discovered and converged. We have extended VLAN 666 across a FabricPath
network across 2 data centers interconnected via OTV!
In the final topology, the traffic flow will be as the following diagram illustrates.
The steps that are taken in the life of the packet are:
2. The packet is encapsulated by SW2 into a FabricPath / Ethernet frame and sent to
the switch where the destination resides.
3. SW1-1 decapsulates the FabricPath header and does a lookup on the destination
address (SW1-4 SVI4) and finds an OTV IP address next-hop in its MAC address-table
4. SW1-1 encapsulates the packet in an OTV / IP frame and sends it across the OTV
network towards the destination
5. SW1-2 decapsulates the OTV header and performs a lookup on the Ethernet frame. The
destination address is a FabricPath Switch ID
7. SW1-4 decapsulates the FabricPath header from the packet and performs a lookup
on the Ethernet frame. The destination interface is it’s local SVI
8. The local SVI receives the Plain Ethernet frame and arrived at its destination.
When ICMP pings between all SVI interfaces in the network are successful the task has been completed
and the chapter has been completed successfully!
The next chapter will be an in-depth overview of NX-OS High Availability features. FabricPath will come
back in this chapter to demonstrate the possibility to connect hosts in a resilient way to 2 leaf switches.
Chapter 4: Data
Center Networking
High Availability
(NX-OS)
Chapter 4: Data Center Networking High Availability (NX-OS) is intended to let you be familiar with the
NX-OS High Availability features on the Nexus platforms to create a high available network. Various
types of deployments of Port-channels and Virtual Port-channels are discussed in this chapter. The
second part of this chapter focuses on First Hop Redundancy Protocols (FHRPs) and High Available
features of dynamic routing protocols. The third part focuses on a special implementation of virtual
port-channels in FabricPath networks.
We highly recommend creating your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab.
General Rules
• Try to diagram out the task. Draw your own connections the way you like it
• Take a very close read of the tasks to ensure you don’t miss any points during grading!
• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Solutions
The layer 3 drawing is in this case more important as the VLANs and SVI’s need to be created out of
this drawing. The drawing shows a number of VLANs that will be running between the routers. Keep in
mind that the layer 2 topology can be very different from the layer 3 topology.
Especially in this chapter we will be creating several layer 2 port-channels (link bundles) that are crossing
all of the switches. Our layer 3 topology however, looks very different than this physical layer 2
topology.
Drawing 2 below illustrates which VLANs and which SVIs need to be created on all switches.
The following configuration is required to finish this initial task of this chapter:
SW1-1
SW1-1(config)# vlan 112, 111, 121, 222
SW1-1(config-vlan)# exit
SW1-1(config)# feature interface-vlan
SW1-1(config)#
SW1-1(config)# interface vlan 112
SW1-1(config-if)# ip address 198.18.112.11/24
SW1-1(config-if)# no shutdown
SW1-1(config)# interface vlan 111
SW1-1(config-if)# ip address 198.18.111.11/24
SW1-1(config-if)# no shutdown
SW1-1(config)# interface vlan 121
SW1-1(config-if)# ip address 198.18.121.11/24
SW1-1(config-if)# no shutdown
SW1-1(config)# interface vlan 222
SW1-1(config-if)# ip address 198.18.222.11/24
SW1-1(config-if)# no shutdown
SW1-1(config)# interface Loopback0
SW1-1(config-if)# ip address 198.18.0.11/32
SW1-2
SW1-2(config)# vlan 123, 111, 121, 222
SW1-2(config-vlan)# exit
SW1-2(config)# feature interface-vlan
SW1-2(config)#
SW1-2(config)# interface vlan 123
SW1-2(config-if)# ip address 198.18.123.12/24
SW1-2(config-if)# no shutdown
SW1-2(config)# interface vlan 111
SW1-2(config-if)# ip address 198.18.111.12/24
SW1-2(config-if)# no shutdown
SW1-2(config)# interface vlan 121
SW1-2(config-if)# ip address 198.18.121.12/24
SW1-2(config-if)# no shutdown
SW1-2(config)# interface vlan 222
SW1-2(config-if)# ip address 198.18.222.12/24
SW1-2(config-if)# no shutdown
SW1-2(config)# interface Loopback0
SW1-2(config-if)# ip address 198.18.0.12/32
SW2
switch(config)# switchname SW2
SW2(config)# vlan 112
SW2(config-vlan)# exit
SW2(config)# feature interface-vlan
SW2(config)#
SW2(config)# interface vlan 112
SW2(config-if)# ip address 198.18.112.2/24
SW2(config-if)# no shutdown
SW2(config)# interface Loopback0
SW2(config-if)# ip address 198.18.0.2/32
Copyright © by IPexpert. All rights reserved. 142
CCIE Data Center Lab Preparation Detailed Solution Guide
SW3
switch(config)# switchname SW3
SW3(config)# vlan 123
SW3(config-vlan)# exit
SW3(config)# feature interface-vlan
SW3(config)#
SW3(config)# interface vlan 123
SW3(config-if)# ip address 198.18.123.3/24
SW3(config-if)# no shutdown
SW3(config)# interface Loopback0
SW3(config-if)# ip address 198.18.0.3/32
After this initial configuration the task is finished and we continue with configuring the layer 2
infrastructure of this topology.
Task 2: Port-Channels
The second task will be about configuring port-channels in a traditional way. The technology for
bundling links has been around for al long time. The technology ensures that multiple physical
connections are bundled into a single logical connection. This bundling is done for multiple reasons:
• Redundancy
o By separating the link bundle across multiple line-cards within a switch, redundancy can
be achieved. When links are removed or added to a bundle, this does not result in loss
of traffic and is therefore a very good failover mechanism.
• Increased bandwidth
o The increased bandwidth is achieved by load balancing traffic over the 2 or more
available connections in the bundle.
• Scalability
o Another reason for this is that when adding links to an existing bundle, it does not result
in downtime. Therefore, a bandwidth capacity increase of connections is very easily
done and does not require a lot of configuration. This makes the connection between
devices very scalable.
• Cost
o When more bandwidth is necessary between 2 devices, you could also upgrade the link
from 1Gbps to 10Gbps for example. However the cost associated with an upgrade to
such a high bandwidth connection is relatively very high, compared to adding a second
1Gbps connection.
o There is always a turning point in which adding more connections to a bundle will cost
more (in OPEX for example) then moving over to a single higher bandwidth connection.
• Loop prevention
o Link bundles are a single connection and therefore there is no reason to run spanning-
tree between the individual members.
The bundling of physical links into a single logical one has many names in different types of
documentation. Vendors also use different naming Some of the names are:
• Port-channels
• EtherChannels
• Trunks (very difficult when people mean different things with the same word)
• VIFs
• (Link) Bundles
• etcetera
Cisco also uses different naming for their link bundles. Within NX-OS everything has been standardized
and is now referred to as Port-Channels. In our workbook we will either be talking about Port-Channels
or Link Bundles.
Link bundling is always done between 2 devices. The logical interface is still a connection between
device A and device B. The drawing below illustrates the bundling of links between 2 devices:
o It’s not necessary that port-channel numbers match on both sides of the connections. In
no protocol these numbers are transported. When you are bundling links between
different vendors switches the numbering is usually different and sometimes not even
possible to match.
o Some hardware platforms might have limitations to the numbering of the link bundles
(older Catalyst switches only supported up to 16 port-channels).
o Interfaces can be mixed and matched across different line cards. Switches have built-in
mechanisms to ensure that packets arrive in the same order (in case of distributed
forwarding) as they arrive on the switch.
There are a number of compatibility requirements for interfaces to come online within a link bundle.
The following list describes the capabilities an interface that need to be matched on all participating
interfaces before they can be added to any bundle:
• Link speed
• Duplex setting
• Layer 3 mode
• Layer 2 mode
• MTU size
• Flow-Control settings
• StormControl settings
The switch will notify you when certain parameters on an interface are incorrect for the switch to add it
to a bundle. The best practice to do is to use a (previously ‘hidden’) command in NX-OS to erase all
interface configurations and reset it to the defaults:
version 5.2(4)
interface Ethernet1/25
switchport
switchport mode trunk
switchport monitor
SW1-1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1-1(config)# default interface Ethernet1/25
SW1-1(config)# sh run int e1/25
version 5.2(4)
interface Ethernet1/25
SW1-1(config)#
When using this command you are sure that the interface will be added to the bundle without any
problems.
Now on to some configuration; the first 2 questions state that a link bundle should be configured
between some switches. It’s best to read ahead on some of the next questions as question 3 and 4 state
some important information to bring up the link bundle.
Port-Channels are given a number to work on. The logical interface which is created uses this number to
be identified throughout the configuration. Numbering is very different per platform. Both platforms
used in our topology (Nexus 7000 and Nexus 5000) support up to 4094 port-channel numbers on the
switch. It’s not said that the switches also supports that amount of port-channels to become
operational, but it’s easier to have your own numbering scheme than to be limited in numbering by the
switch capabilities.
When no number is given, just use your own best practice for it. Ensure that you do not overlap on the
same switch and I would use the same number on both sides of the link so you never make mistakes
when configuring certain port-channels on different devices.
We will first configure the link bundle between SW1-2 and SW3 as we require additional configuration
for the first question.
SW1-2
SW1-2(config)# default interface Ethernet3/5
SW1-2(config)# default interface Ethernet3/6
SW1-2(config)# interface e3/5-6
version 5.2(4)
interface Ethernet3/5
switchport
switchport mode trunk
switchport trunk allowed vlan 123
channel-group 2
no shutdown
version 5.2(4)
interface Ethernet3/6
switchport
switchport mode trunk
switchport trunk allowed vlan 123
channel-group 2
no shutdown
version 5.2(4)
interface port-channel2
switchport
switchport mode trunk
switchport trunk allowed vlan 123
SW1-2(config-if)#
As there is no information about any port-channeling protocol, we will use no protocol and just use the
static port-channeling configuration. Now one side of the connection is configured.
The interfaces do not use any port-channeling protocol as mentioned, but are statically configured
inside of a port-channel. This means that when both links are up. The interfaces are forced into a port-
channel.
Keep in mind that this might cause strange behaviors on the other side of the connection as it will not
expect traffic to be load balancing between 2 individual links. This means that you might see errors for
MAC address flapping or even get ports that are becoming error disabled (err-disabled) because of a
potential loop in the network.
SW3
SW3(config)# default interface Ethernet1/1
SW3(config)# default interface Ethernet1/2
SW3(config)# interface e1/1-2
SW3(config-if-range)# channel-group 2 mode on
SW3(config-if-range)# no shutdown
SW3(config-if-range)# interface port-channel2
SW3(config-if)# switchport
SW3(config-if)# sw mode trunk
SW3(config-if)# sw trunk allowed vlan 123
SW3(config-if)# no shut
Now both sides of the connection are configured and therefore the port-channel has successfully come
online and the SVI interfaces can ping each other on the switches.
SW1-2(config-if-range)#
Next up is the first connection. We want to configure port-channel 1 with some special parameters.
These include that the bundle should be negotiated between the switches before it comes online. This is
related to port-channel protocols like PAgP and LACP. The first protocol is an older Cisco proprietary
protocol which is no longer supported on NX-OS platforms. The standards based solution is the Link
Aggregation Control Protocol (LACP). This protocol uses a hello mechanism to identify
certain interfaces and switches between each other. When every capability is correctly advertised
between the switches the port-channel interfaces will come online. The most important thing a switch
negotiates with LACP is the system ID. This ID needs to be advertised on both of the interfaces. This
ensures that the switch is connected to one device only and that there is no loop in the network.
The hello mechanism also helps in redundancy switchovers. With a static port-channel it is required for
the interface to go down. Otherwise the link will remain in the bundle. With LACP an intelligent
mechanism ensures that when a certain amount of hello packets is missed, according to a threshold, the
link is removed from the bundle. This makes it possible that when there is equipment between the 2
switches which will not propagate link down events, the switch will still detect this by the missing LACP
packets.
In the following drawing a few scenarios are described that would cause errors in a configuration with
static port-channels, but is automatically solved when using the LACP protocol.
o Dependent on the configuration of the OS, links that are configured in a LACP port-
channel on only 1 side of the connection. They can still become online in an
Individual mode (no bundling) without any port-channeling capabilities. This is
different on NX-OS where by default links are disabled.
• Hot-standby interfaces
o When interfaces are inserted into the link bundle that go over the maximum amount. A
static port-channel would reject the configuration, but in an LACP configuration it can
still be accepted (Still dependent on other maximums). When one of the active
interfaces than fails the now ‘Hot-Standby’ interface will become inserted into the
bundle, so that in case of a loss of link, there is no capacity decrease in the link bundle.
o When one interface is placed in one bundle, where the other device is configured for
another configuration of bundles (like the bottom links in the drawing). The LACP port-
channel will go in a Suspended state, where as a static port-channel will have serious
problems!
Now on to the configuration of our LACP port-channel. The fourth question states that SW2 should
never start the negotiation of the port-channel. This refers to the 2 modes of LACP port-channeling that
can be configured. With all above information we can summarize in the 3 different modes that port-
channels can be configured in.
Mode Description
On The default configuration. This mode does not initiate LACP messages on the port-
channel. This mode is also referred to as ‘static’ port-channels. Interfaces that
come online are immediately placed
Active This mode enables the LACP protocol on this port-channel. After the interface
comes online, it immediately starts sending LACP packets across the link.
Passive This mode also enables the LACP protocol on a port-channel. The difference with
active mode is that in this passive mode the switch waits for another LACP packet to
be received on the interface, before it replies back with an LACP message
This results in the following table of modes. Only the yellow marked fields will result in an operational
port-channel. The other mode combinations will result in a Suspended or Non functional port-channel.
On Active Passive
On Yes No No
Passive No Yes No
Now on to the configuration of our second port-channel. SW1-1 and SW2 will be configuring a port-
channel with ID 1 between each other.
SW1-1
SW1-1(config)# feature lacp
SW1-1(config)# default interface e3/1
SW1-1(config)# default interface e3/2
SW1-1(config)# int e3/1-2
SW1-1(config-if-range)# channel-group 1 mode active
SW1-1(config-if-range)# no shut
SW1-1(config-if-range)# int po1
SW1-1(config-if)# sw
SW2 is stated to never negotiate the interface bundle. Therefore we will use the Passive mode
of the LACP port-channeling. After the configuration the port-channel comes online and the SVI
is working correctly.
SW1-1(config-if)# sh port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------
------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------
------
1 Po1(SU) Eth LACP Eth3/1(P) Eth3/2(P)
The next two questions in this task are regarding minimum and maximum interfaces in a port-channel.
The second question is regarding the hot-standby feature.
For various reasons it’s possible that you want to steer the amount of interfaces that are up in the port-
channel. One of the most important reasons is that you have a network topology where the port-
channels are not used for redundancy.
In the case which is illustrated in the drawing above, you would not want the port-channel to remain up
in this case. Imagine that the left-side path is chosen in this topology and one of the interfaces fails while
there is 16Gbps of traffic crossing the links. Now there would only be 10Gbps available on that
connection. The link cost/metric is not changed when a link fails and therefore dynamic routing
protocols are not aware of any changes in the network.
The 16Gbps of traffic will still be sent across the interface that now only has 10Gbps of bandwidth. In
this case you will want the network to failover to the right-side path of the drawing so the traffic can still
be sent to where it needs to go, across a 20Gbps connection.
SW1-1
SW1-1(config)# interface Port-channel1
SW1-1(config-if)# lacp min-links 2
SW2
SW2(config)# interface Port-channel1
SW2(config-if)# lacp min-links 2
Now when this feature is configured and we disable one of the member links in the port-channel. We
see that immediately the interface is disabled and no longer working.
SW1-1
SW1-1(config)# interface Ethernet3/1
SW1-1(config-if)# shutdown
SW2
SW2(config)#
The interface went down with a clear indication that this is because the minimum amount of links that is
configured (Default is 1) is not met in this case.
The next question is regarding the maximum amount of interfaces that may be placed into a bundle.
This command is not available in a static port-channel. The maximum amount of interfaces that can be
enabled on a Nexus 7000 or Nexus 5000 is 8 active interfaces.
When you go over this amount and try to activate a 9th link in a static bundle. It will reject the command
and will not place the link into the bundle. When this same task is done on an LACP bundle, the
command is accepted and the link is placed into a ‘hot-standby‘ state. This means that when one of
the active links fail in the bundle, the hot-standby link will take its place and will become active in the
link bundle.
The maximum amount of active interfaces in a link bundle remains 8, also with LACP port-channels.
Then the maximum amount of hot-standby interfaces is also 8. This means that the maximum
amount of interfaces that can be configured within a LACP port-channel is 16.
SW1-1
SW1-1(config)# interface Port-channel1
SW1-1(config-if)# lacp max-bundle 2
SW2
SW2(config)# interface Port-channel1
SW2(config-if)# lacp max-bundle 2
SW2
SW2(config)# interface Ethernet1/3
SW2(config-if)# channel-group 1 mode passive
We see that this is accepted, but the interface is placed in hot-standby mode, because the maximum
amount of interfaces is set to 2.
The next question is regarding a new port-channel between SW2 and SW3. This interface is not used as a
layer 2 connection. This one should be configured as a Layer 3 routed interface. Last thing is that
the interface should be negotiated by both sides. In an earlier question it was stated that SW2 should
never be actively negotiating it’s connection. So SW2 should be set on passive mode.
In production networks it’s a best practice to always configure both sides to be active.
SW2
SW2(config)# default interface e1/5
SW2(config)# default interface e1/6
SW2(config)# int e1/5-6
SW2(config-if-range)# channel-group 3 mode passive
SW2(config-if-range)# no shut
SW2(config-if-range)# int po3
SW2(config-if)# no switchport
SW2(config-if)# ip address 198.18.23.2/24
SW2(config-if)# no shut
SW3
SW3(config)# default interface e1/5
SW3(config)# default interface e1/6
SW3(config)# int e1/5-6
SW3(config-if-range)# channel-group 3 mode active
SW3(config-if-range)# no shut
SW3(config-if-range)# int po3
SW3(config-if)# no switchport
SW3(config-if)# ip address 198.18.23.3/24
SW3(config-if)# no shut
The next question is referring to a behavior that has changed in NX-OS compared to Classical IOS. In
Classical IOS when an interface that was brought up in a LACP port-channel without any configuration on
the other end of the link. Meaning that it would not receive any LACP packets from the other device.
The link would still become available, but in a so-called ‘Individual’ state. This means the link would
become available in a normal link state, unbundled. The Port-channel interface would not see this
interface as a member until it receives LACP packets.
Within NX-OS this behavior changed. When the interface comes online on an NX-OS interface by default
the interface will be in Suspended mode and not come online. This is the better behavior as the
Individual interfaces might cause issues and can even cause loops.
SW2
SW2(config-if)# int po3
SW2(config-if)# no lacp suspend-individual
ERROR: Cannot set/reset lacp suspend-individual for port-channel3 that is
admin up
SW2(config-if)# shut
SW2(config-if)# no lacp suspend-individual
Warning: !! Disable lacp suspend-individual only on port-channel with edge
ports. Disabling this on network port port-channel could lead to loops.!
SW2(config-if)# no shut
SW2(config-if)#
The switch already warns you that the Individual state might cause loops in the network, so the
best practice is to leave this enabled!
SW3
SW3(config-if)# int po3
SW3(config-if)# shut
SW3(config-if)# no lacp suspend-individual
Warning: !! Disable lacp suspend-individual only on port-channel with edge
ports. Disabling this on network port port-channel could lead to loops.!
SW3(config-if)# no shut
SW3(config-if)#
To test the behavior we will disable the LACP on one of the links and see what happens.
On the other side of the link, the link went into suspended mode.
When the link is configured like stated in question 9. The interfaces are up in an individual state
and the port-channel remains down.
Question 10 relates to the selection of ports within a Port-Channel. This selection is crucial for which
interfaces become Participating in a bundle and which ones become hot-standby. It has also
to do which interface is selected as the ‘Primary’ interface in a bundle, where the major amounts of
LACP packets are sent through.
As the maximum amount of links in a bundle is 8, the maximum capacity of the bundle is 80Gbps.
This question states that one of the links should always be chosen to participate when it’s active and
one should be selected as hot-standby immediately after new interfaces are added to the bundle.
The mechanism that is used for this task is ‘LACP port-priority’. This is a similar priority value to
the Spanning-Tree port-priority value where a port on a local switch is selected. This value is never
transported outside of the switch.
The lower the value that is configured. The higher priority is assigned to an interface. Therefore we give
Ethernet1/5 a very low value (lowest possible) to ensure that it is always chosen.
SW2
SW2(config-if)# int Ethernet1/5
SW2(config-if)# lacp port-priority 1
SW3
SW3(config-if)# int Ethernet1/5
SW3(config-if)# lacp port-priority 1
The default priority value is 32768. This means that if we want the interface to become hot-standby as
soon as an additional interface is added to the link-bundle this value should be increased.
SW2
SW2(config-if)# int Ethernet1/6
SW2(config-if)# lacp port-priority 65000
SW3
SW3(config-if)# int Ethernet1/6
SW3(config-if)# lacp port-priority 65000
The next question states that Port-channel3 should be using a very fast detection mechanism. Within
NX-OS it’s not possible to change the LACP timers by configuring specific values. It is possible however to
use sub-second hello messages and have a timeout timer of 1 second to ensure that a neighbor is
signaled as down after 1 second.
This is done under individual interfaces and the interface needs to be disabled before this is turned on!
SW2
SW2(config-if)# int e1/5
SW2(config-if)# lacp ?
port-priority Set LACP port priority
rate Configure rate at which PDUs are sent by LACP
SW3
SW3(config-if)# int e1/5
SW3(config-if)# shut
SW3(config-if)# lacp rate fast
SW3(config-if)# no shutdown
SW3(config-if)# int e1/6
SW3(config-if)# shut
SW3(config-if)# lacp rate fast
SW3(config-if)# no shut
The next question requires the configuration of load balancing parameters. This is a global setting for
the entire switch. It can also be configured per line card. However there are very few cases in which this
would be recommended. The global setting is relevant for outgoing traffic only. There is not option to
control the way traffic is incoming on the switch itself.
The load-balancing can be a crucial setting on the switch, but is very dependent on what is connected to
the port-channels that are configured on the switch. The configuration is dependent on the hardware
platforms. Some platforms might not support all types of load balancing. Within the NX-OS platform,
they are all based on high-end switching platforms. So both the Nexus 7000 and the Nexus 5000
platforms in our topology support all of the options.
The load balancing is accomplished using a ‘hash’ of various kinds of information that is found in the
header of a packet. This hash is generated through a combination of multiple values found in the Layer
2, Layer 3 and/or Layer 4 header.
The load-balancing options are separated in multiple categories. For layer 2 ethernet traffic a different
load balancing is applied than for layer 3 IP traffic.
• Destination IP address
• Source IP address
The NX-OS platform does not care when traffic that is sent on one interface and received on another
interface. Therefore it’s not required to have an equal setting on both sides of the interfaces, but it is
recommended to have it equal.
A simple mathematic calculation can be done that there when more fields are taken into the hash, the
more different hashes are generated. This means in a more granular load balancing. The load balancing
based on fields in the IP header also takes care of that all traffic that is related to a single traffic flow is
always sent on a single interface. This overcomes the issue when the approach would be a per-packet
load balancing scheme that traffic can arrive in a different order.
Having the traffic flows sent across a single interface helps in the performance of the application.
In case of layer 3 connections there is no reason to do load balancing across MAC addresses, because
the routed interfaces are the only 2 MAC addresses that are used in the IP subnet. Therefore these
options are not available in the Layer 3 load balancing options.
To complete this task it’s stated that the most information should be used for load balancing traffic
across the member links in a port-channel.
On the Nexus 7000 this command cannot be issued from a Virtual Device Context (VDC). Only the
default VDC can change the load balancing algorithm for the switch and modules. In our question we
only need to change the Nexus 5000 switches.
Remember that layer 2 and layer 3 load balancing needs to be changed as we have both layer 2 and
layer 3 Port-Channels on these switches.
SW2
SW2(config)# show port-channel load-balance
IP: src-dst ip
First we configure the Layer 2 Ethernet load balancing using the ‘ethernet’
statement.
SW2(config)#
SW2(config)#
SW2(config)# port-channel load-balance ?
dst Destination based parameters
ethernet Ethernet port-channel
src Source based parameters
src-dst Source-destination based parameters
The other configuration options (src, dst, src-dst) are used for the Layer 3 IP load-balancing.
SW3
SW3(config)# port-channel load-balance ethernet source-dest-ip-port-vlan
SW3(config)# port-channel load-balance src-dst ip-l4port-vlan
We verify the new configuration and now we are using the best load balancing possible.
SW2(config)#
Before we can continue with the next task we need to remove the port-channel configuration between
the switches as we will be re-using those interconnecting links for exciting new virtual Port-Channels!
SW1-1
SW1-1(config)# no interface port-channel 1
SW1-1(config)# default interface Ethernet3/1
SW1-1(config)# default interface Ethernet3/2
SW1-2
SW1-2(config)# no interface port-channel 2
SW1-2(config)# default interface Ethernet3/5
SW1-2(config)# default interface Ethernet3/6
SW2
SW2(config)# no interface port-channel 1
SW2(config)# default interface Ethernet1/1
SW2(config)# default interface Ethernet1/2
SW3
SW3(config)# no interface port-channel 2
SW3(config)# default interface Ethernet1/1
SW3(config)# default interface Ethernet1/2
This technology has been engineered to solve a very common problem. One of the reasons to
implement Port-Channels is to deliver redundancy to the interfaces. As stated in the previous task the
failover to a remaining interface or the insertion of a new link in the Port-Channel does not result in any
traffic loss. This ensures a very smooth operational scenario where links can be easily expanded and
failover occurs without any impact.
The redundancy however is not well engineered as the link is still made between 2 single equipped
devices. If one of the 2 (core) devices fails, the entire link goes down as illustrated by the drawing below.
The connecting device needs a manual switchover to the other link or another failure mechanism like
Spanning-Tree to use the other link.
To overcome this redundancy issue we want to use a technology to be able to create a Port-Channel
between 2 different switches that simulate to be one logical switch with one logical link between the
host/switch and itself.
There are multiple names for this technology, all resulting in the same basic principle. A few examples
are:
One generic vendor-independent term used in this case is Multi-Chassis Link Aggregation Groups (MC-
LAG).
The Virtual Port-Channeling technology is the only technology that leverages a different approach than
other implementations of MC-LAG. The vPC technology is the only one that uses separated control-
planes for both switches. In other implementations the control-planes are merged so you only have a
single management entity to mange.
In case of the vPC technology you keep 2 separate switches (control-planes) that have shared some
features with each other. This has a very special reason why this is done on the Nexus series. The Nexus
series are also capable of transferring FCoE traffic, which is explained in more detail in a later chapter.
The Fibre-Channel protocol requires the separation of 2 ‘fabrics’. The traffic that uses Fabric A is never
allowed to cross any devices that are related to Fabric B. For this reason the vPC technology does not
combine the control-planes of the switches so you can keep your Fibre Channel (oE) fabrics separated!
The technology uses a number of components to be able to function correctly and to ensure a loop free
topology for the vPC connected devices.
A vPC identifies itself towards and end-host as a normal LACP or Static port-channel. The end device has
no indication that it is connected to 2 different switches and therefore assumes that it’s a normal Port-
Channel. The technology only operates on the Nexus switches. Before a vPC will come online a number
of topics need to be in place:
o The vPC domain configuration configures the role of the device (primary / secondary)
and the Domain ID.
o The Domain ID needs to match on both devices that are able to create vPCs between
each other
• vPC Peer-Link
o The vPC peer keepalive link is also a very crucial part of the set-up. Without a dedicated
keep-alive link the vPCs will not come online. This link is used for simple hello packets
(can be send across a layer 2 or a layer 3 network).
o The Hello packets are required for split-brain detection, in case the Peer-Lin k
fails, one of the vPC peer devices (the secondary) will shut down its vPC ports
o You require a dedicated GigabitEthernet interface or you can use the out-of-band
management Ethernet port (mgmt0)
The drawing below illustrates the terminology of the mentioned required topics:
All topics are discussed in the questions in this task. The initial question is to ensure that SW1-1 and
SW1-2 will form a pair to create vPC’s. Again it’s good to read ahead as we require some additional
information regarding the set-up of our peer-link and peer keepalive link.
We need all this information before the vPC’s will be able to form and before we will see information
regarding the switches forming an adjacency.
The information that is exchanged between the vPC switches uses the Cisco Fabric Services over IP
protocol (CFS). This protocol has been used by the Cisco MDS (fibre channel) switches for years to
exchange information over the Fibre Channel protocol about the other switches. Using this protocol
Cisco has been able to extend the Fibre Channel features with this synchronization protocol. The IP
version of this protocol has been extended to support vPC’s on the Nexus series.
There are some limitations on the vPC technology. These are a few of the most important limitations:
o You can’t mix Nexus 5000 and Nexus 7000 switches to be in the same vPC domain
o On the Nexus 7000 you cannot mix line card types. When the first switches uses M1 line
cards, the second switch should also use an M1 line card for both the vPC Peer Link and
the vPC client interfaces
Copyright © by IPexpert. All rights reserved. 168
CCIE Data Center Lab Preparation Detailed Solution Guide
Now on to some configuration for the 2 domains that we are going to configure.
We enable the vPC functionality first and need to ensure that SW1-1 is the Primary switch in the vPC
configuration and SW1-2 the secondary.
SW1-1
SW1-1(config)# feature vpc
SW1-1(config)# vpc domain 100
SW1-1(config-vpc-domain)# role ?
priority Configure priority to be used during vPC role
(primary/secondary)
election
SW1-2
SW1-2(config)# feature vpc
SW1-2(config)# vpc domain 100
SW1-1(config-vpc-domain)# role priority 1000
Warning:
!!:: vPCs will be flapped on current primary vPC switch while attempting
role change ::!!
Note:
--------:: Change will take effect after user has re-initd the vPC peer-
link ::--------
We use the Role Priority command to determine which switch will be the primary and which switch will
be the secondary switch in our vPC configuration. Like the switch mentions. When you are doing this in
an active configuration. The peer-link should be flapped to ensure the new roles are assigned. This has
to do with the nature of the role. It’s not pre-empted, meaning that it will not flap when it sees a peer
come online with a better priority value. Only after re-initialization the roles are (re-)assigned. In this
case we make sure that SW1-1 has a much lower priority value (and therefore a higher priority) than
SW1-2.
Copyright © by IPexpert. All rights reserved. 169
CCIE Data Center Lab Preparation Detailed Solution Guide
On SW2 and SW3 we do not care about the role of the devices in the vPC configuration. Remember that
when you are configuring vPC’s between 2 different sets of Nexus switches like in our topology. You
need to use different Domain ID’s. Otherwise the vPC’s will not come online when equal Domain ID’s
are used between the 2 sets of switches.
SW2
SW2(config)# feature vpc
SW2(config)# vpc domain 200
SW3
SW3(config)# feature vpc
SW3(config)# vpc domain 200
Next up are the configurations for the peer keep alive links. These should always be configured before
you go to the vPC Peer-link configuration. The vPC Peer-link will not come online when no hello
messages are received over the peer keep alive link. This link is crucial in the set-up!
On the Nexus 5000 switches we will be using the mgmt0 interface. On the Nexus 7000 it’s the best
practice to use a dedicated GigabitEthernet interface from the line cards instead of the management
interface on the supervisor.
SW2
SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3 vrf
management
SW3(config-vpc-domain)#
SW3
SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2 vrf
management
SW3(config-vpc-domain)#
Remember to always specify the VRF that is used for these keep alive messages. As a best practice they
should never be sent from the Global Routing Table. On the Nexus 5000 we use the mgmt0 interface for
our keepalives.
After configuration, the vPC protocol has not been successfully established, but we do see another peer
and mention that it is alive!
The next question is about configuring the vPC peer keepalives across a dedicated SVI and the traffic
should be separated, like the best practice, from the global routing table. We do this by using a
dedicated VRF for the keepalives. The question did not specify any VLAN number, so we can use
anything we want. As all other IP subnets keep their VLAN number equal to the third octet, we will keep
this standardization to make it easier for ourselves.
We configure the SVI and VRF first and then configure the peer-keepalive statement:
SW1-2
SW1-2(config)# int e3/12
SW1-2(config-if)# sw mode trunk
SW1-2(config-if)# sw trunk allowed vlan 5
SW1-2(config-if)# no shut
SW1-2(config-if)# vlan 5
SW1-2(config-vlan)# exit
SW1-2(config)# feature interface-vlan
SW1-2(config)# int vlan 5
SW1-2(config-if)# ip add 198.18.5.12/24
SW1-2(config-if)# no shut
SW1-2(config-if)# ping 198.18.5.11
PING 198.18.5.11 (198.18.5.11): 56 data bytes
Request 0 timed out
64 bytes from 198.18.5.11: icmp_seq=1 ttl=254 time=1.518 ms
64 bytes from 198.18.5.11: icmp_seq=2 ttl=254 time=0.825 ms
64 bytes from 198.18.5.11: icmp_seq=3 ttl=254 time=0.838 ms
64 bytes from 198.18.5.11: icmp_seq=4 ttl=254 time=0.731 ms
We can ping the other end successfully, but we also need to configure the VRF.
Be aware that all IP addressing and other Layer 3 configuration is removed from the interface once it’s
placed into a VRF!
By default the management IP address is used as the source address for the vPC keep alives. Therefore
it’s required to configure the source IP address when you are using another interface than the mgmt0.
SW1-1
SW1-1(config)# int e3/10
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# sw trunk allowed vlan 5
SW1-1(config-if)# no shut
SW1-1(config-if)# vlan 5
SW1-1(config-vlan)# exit
SW1-1(config)# feature interface-vlan
SW1-1(config-if)# vrf context keepalives
SW1-1(config-vrf)# int vlan 5
SW1-1(config-if)# vrf member keepalives
SW1-1(config-if)# ip add 198.18.5.11/24
SW1-1(config-if)# no shut
SW1-1(config-if)# vpc domain 100
SW1-2(config-vpc-domain)# peer-keepalive destination 198.18.5.12 vrf
keepalives source 198.18.5.11
SW1-1(config-vpc-domain)#
The last required configuration before we can bring up the vPC’s itself are the vPC peer-links. In one
example we will be using a port-channel (standard) to ensure that a link can fail within the peer-link
before it’s resulting in erroneous operational issues. This is however not required and we can also
configure just a single link as the vPC peer-link.
Remember that the peer-link might also carry data traffic, so scaling the link in terms of uplink capacity
is very important.
First we create a single link peer-link between SW1-1 and SW1-2. The peer-link should either be a
FabricPath enabled interface or a Layer 2 trunk interface with all VLANs enabled.
SW1-1
SW1-1(config)# int e3/9
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# vpc peer-link
SW1-1(config-if)# no shut
SW1-2
SW1-2(config)# int e3/11
SW1-2(config-if)# sw mode trunk
SW1-2(config-if)# vpc peer-link
SW1-2(config-if)# no shut
The second example is configuring the peer-link as a port-channel. We first create a standard LACP
based port-channel between the 2 switches and enable the vPC peer-link functionality on these
interfaces.
SW2
SW2(config)# int e1/7-8
SW2(config-if-range)# channel-group 23 mode active
SW2(config-if-range)# exit
SW2(config)# interface port-channel23
SW2(config-if)# sw mode trunk
SW2(config-if)# vpc peer-link
SW2(config-if)# no shut
SW3
SW3(config)# int e1/7-8
SW3(config-if-range)# channel-group 23 mode active
SW3(config-if-range)# exit
SW3(config)# interface port-channel23
SW3(config-if)# sw mode trunk
SW3(config-if)# vpc peer-link
SW3(config-if)# no shut
After the configuration the vPC peer-link is signaled up and working and the switches are ready to be
configuring vPC port-channels!
The next question asks to ensure that the vPC’s remain up and running when a peer fails or reboots. The
split-brain detection works in a way that the secondary switch turns it’s vPCs off when it suspects
a split-brain. To recover from this you can force the switch to bring up its VDCs after a time-out.
SW1-1
SW1-1(config-if)# vpc domain 100
SW1-1(config-vpc-domain)# auto-recovery ?
<CR>
reload-delay Duration to wait before assuming peer dead and restoring
vpcs
SW1-2
SW1-2(config-if)# vpc domain 100
SW1-2(config-vpc-domain)# auto-recovery reload-delay 300
Remember that there are 2 features that do almost the same. Before NX-OS 5.2(1) the command
used to solve this was ‘reload restore’. After this release it has been deprecated and replaced with
auto-recovery.
The reload restore command did not solve the issue when a peer failed, only in case of a reboot.
Therefore in this question the only right answer is the new version of the command.
The spanning-tree question is related to vPC configuration where the vPC switches will identify
themselves as a single Spanning-Tree switch to the connecting switches that think they are connecting
with a single switch. Therefore the spanning-tree configuration that is made locally only applies to that
Copyright © by IPexpert. All rights reserved. 174
CCIE Data Center Lab Preparation Detailed Solution Guide
specific switch and not to connecting switches that are using vPCs. Except when the ‘peer-switch’
configuration is applied. Then the CFS protocol is used to push the spanning-tree control plane
messages to the other switch. You still need to configure the spanning-tree settings independently, but
when they match. The switch will be identified as a single root bridge towards the rest of the network.
SW2
SW2(config-if)# vpc domain 200
SW2(config-vpc-domain)# peer-switch
SW3
SW3(config-if)# vpc domain 200
SW3(config-vpc-domain)# peer-switch
Now to verify we check the spanning-tree output of VLAN 1 for both switches. We see a default priority
value is set.
SW2(config)# sh spanning-tree
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 32769
Address 0023.04ee.be64
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
SW3# sh spanning-tree
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 32769
Address 0023.04ee.be64
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
N7K-8-1#
Now we change the priority to the asked 8192 and check again.
SW2
SW2(config-if)# spanning-tree vlan 1 prio 8192
SW3
SW3(config-if)# spanning-tree vlan 1 prio 8192
SW2(config)# sh span
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 8193
Address 0023.04ee.be64
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
SW3(config)# sh span
VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 8193
Address 0023.04ee.be64
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Although the switches both identify as root. SW3 also has a root port as SW2 is the true root bridge in
this network. Still SW3 will recognize itself as the root bridge and will act like it for the rest of the
network!
The next task is one where we finally begin configuring the vPCs!
The initial connection is made between vPC domain 100 and a ‘client’ switch, which is SW2 in this case.
From SW2’s perspective this is a standard port-channel towards another standard switch.
We first configure the port-channel on SW2. We use LACP in this case, but we could’ve used a static
Port-Channel (mode on) as well. The best practice however is to use LACP port-channels.
SW2
SW2(config-if)# int e1/1-2
SW2(config-if-range)# channel-group 101 mode active
SW2(config-if-range)# no shut
SW2(config-if-range)#
SW2(config-if-range)# int port-channel101
SW2(config-if)# sw mode trunk
SW2(config-if)# no shut
SW2(config-if)# sh port-chan summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------
------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------
------
101 Po101(SD) Eth LACP Eth1/1(I) Eth1/5(I)
SW2(config-if)#
The port-channel is currently not operational because we haven’t configured the other side yet.
SW1-1
SW1-1(config-if)# int e3/1
SW1-1(config-if)# channel-group 101 mode active
SW1-1(config-if)# no shut
SW1-1(config-if)# int port-channel101
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# vpc 101
SW1-1(config-if)# no shut
SW1-1(config-if)#
SW1-2
SW1-2(config-if)# int e3/3
SW1-2(config-if)# channel-group 101 mode active
SW1-2(config-if)# no shut
SW1-2(config-if)# int port-channel101
SW1-2(config-if)# sw mode trunk
SW1-2(config-if)# vpc 101
SW1-2(config-if)# no shut
Copyright © by IPexpert. All rights reserved. 177
CCIE Data Center Lab Preparation Detailed Solution Guide
SW1-2(config-if)#
The only thing specific about this configuration is the ‘vpc’ command. This command needs to be the
same on both vPC peer switches. Port-channel numbers do not need to match, but vpc numbers do! It’s
highly recommended to keep the port-channel number and vpc number equal, because it eases your
troubleshooting a lot! There is also no reason why these should be different.
After configuring the vPC peer switches we see the port-channel (standard) is now up and running on
SW2.
On the vPC switches we also see a perfectly functioning port-channel, except with just one member.
Channel
--------------------------------------------------------------------------
------
101 Po101(SU) Eth LACP Eth3/3(P)
The most important verification command is to see if the vPC has been formed OK and if the state
information is correctly synced between the switches.
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
101 Po101 up success success 1
The vPC is up and running! A lot of things can go wrong in your configuration before the vPC is up and
running. For example, the allowed VLAN list should be 100% equal on both sides, or the vPC will stay
down with an error message. The ‘show vpc’ command will show you a lot of information regarding
the reason of the vPC being down. Usually it’s very easily spotted when you issue a show running for
both interfaces and see that there are differences in the configuration.
The next vPC that needs to be configured is very much the same as the previous one. Except that this
one is using vPC domain 200 instead of 100. This means that both Nexus 5000s are used as vPC peers
and the Nexus 7000 is the client switch where the vPC is connected to.
The configuration is straight forwards and a good practice for another vPC connection:
SW1-2
SW1-2(config-if)# int e3/5,e3/7
SW1-2(config-if)# channel-group 102 mode on
SW1-2(config-if)# no shut
SW1-2(config-if)# int port-channel102
SW1-2(config-if)# sw mode trunk
SW1-2(config-if)# no shut
SW1-2(config-if)#
SW2
SW2(config-if)# int e1/3
SW2(config-if)# channel-group 102 mode on
SW2(config-if)# no shut
SW2(config-if)# int port-channel102
SW2(config-if)# sw mode trunk
SW2(config-if)# vpc 102
SW2(config-if)# no shut
SW2(config-if)#
SW3
SW3(config-if)# int e1/3
SW3(config-if)# channel-group 102 mode on
SW3(config-if)# no shut
SW3(config-if)# int port-channel102
SW3(config-if)# sw mode trunk
SW3(config-if)# vpc 102
SW3(config-if)# no shut
One important command to check is the following command, giving an overview of all settings that are
checked on the ports before the vPC comes online.
Legend:
Type 1 : vPC will be suspended in case of mismatch
When we, for example, change the Native VLAN on the port-channel on one of the vPC peers. The vPC is
brought down, because traffic is no longer guaranteed to be treated as it should on each of the vPC
peers. This check is pro-active and happens right after the command has been issued.
SW2
SW2(config-if)# interface port-channel102
SW2(config-if)# switchport trunk native vlan 5
Legend:
Type 1 : vPC will be suspended in case of mismatch
SW2(config-if)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC status
--------------------------------------------------------------------------
--
id Port Status Consistency Reason Active
vlans
------ ----------- ------ ----------- -------------------------- ---------
--
102 Po102 down* failed Compatibility check failed -
for native vlan
Like previously mentioned. The show vpc shows already most important information.
In a more recent NX-OS release, it became possible to have the vPC come online even when there is a
consistency error. The graceful consistency-check enables the interface of the vPC
terminating on the Primary vPC peer device to come online and have the interface terminating on the
Secondary peer go in suspended mode. This way the traffic will not have impact on a configuration
error.
You could’ve used an LACP channel as well, as the question did not indicate which way to use. To
demonstrate that static port-channels also work in this configuration this was used in the example.
The next task is to use the remaining connections to finish the vPC configurations. These connections
that remain will be a vPC that is connected to another vPC. For the switches itself they will look like
standard vPC port-channels, but they are what is called ‘back-to-back-vPCs’.
This single connection is definitely a best practice. The important part of this is that the vPC Domain ID’s
not never overlap in this configuration. This can lead to very strange situations.
SW1-1
SW1-1(config-if)# int e3/2,e3/4,e3/6
SW1-1(config-if)# channel-group 103 mode active
SW1-1(config-if)# no shut
SW1-1(config-if)# int port-channel103
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# vpc 103
SW1-1(config-if)# no shut
SW1-1(config-if)#
SW1-2
SW1-2(config-if)# int e3/5
SW1-2(config-if)# channel-group 103 mode on
SW1-2(config-if)# no shut
SW1-2(config-if)# int port-channel103
SW3
SW3(config-if)# int e1/1-2,e1/4
SW3(config-if)# channel-group 103 mode on
SW3(config-if)# no shut
SW3(config-if)# int port-channel103
SW3(config-if)# sw mode trunk
SW3(config-if)# vpc 103
SW3(config-if)# no shut
The next question in which we are asked to allow all VLANs required for our Layer 3 topology to be
traveling the vPC links is already completed as we did not place an VLAN allowed list on the vPCs. So we
can just use them for the next tasks.
The final task is to let the vPC domain 100 identify itself through LACP using a specific MAC address.
One reason for LACP port-channels to work is, that they ensure loop prevention. When different
switches are seen on interfaces the port-channel will not come online. This would cause issues on vPC
enabled switches as they still have 2 separate control planes and therefore different MAC addresses. To
be able for LACP port-channels to work. The vPC switch ‘spoofs’ a single MAC address of one of the vPC
peer members for this.
The field we are looking for in the LACP packet is the System-MAC field. This is changed in the vPC
domain configuration.
SW1-1
SW1-1(config-if)# vpc domain 100
SW1-1(config-vpc-domain)# system-mac 1234.5678.90ab
Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no]
y
SW1-2
SW1-2(config-if)# vpc domain 100
SW1-2(config-vpc-domain)# system-mac 1234.5678.90ab
Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no]
y
As the command already mentions. The vPC peer-link and vPC’s will flap when this command is issued
and you have LACP based port-channels running on the vPC enabled switch.
The Graceful Restart functionality is a process that works in dynamic routing protocols. It only
works on platforms that have a clear separation between control-plane and data-plane. The technology
requires the support of the dynamic routing protocols on which it is enabled. These protocols are able
to signal a special packet once the router switches over to another Supervisor (Route Processor, Route
Switch Processor, Routing Engine, etc.) or when they will completely reboot.
By signaling this special packet. The neighboring routers will know that a router is undergoing a change
(software upgrade, reboot, switchover, etc.). They will keep the routing information about this neighbor
in their respective routing tables and signal the neighbor down according to standard time-outs. The
traffic can still be directed towards the neighbor who is rebooting, because the data/forwarding plane is
not dependent on the control-plane to operate. Therefore traffic can still be sent to it.
When the router is rebooted / switched over, the dynamic routing protocol adjacencies are established
again. Routes are being learned using the normal process and the router is back up and running. Traffic
had no impact on this change.
The routing protocols that are supported on the NX-OS devices like OSPF and EIGRP are by default
enabled for graceful-restart. The feature will only work when both neighbors advertise the capability to
each other. So it’s safe to enable the feature, if the other neighbor doesn’t support it, the feature is not
used.
In this task we first need to configure some dynamic routing protocols. We will use the routing
information configured now in later tasks!
SW1-1
SW1-1(config-if)# feature ospf
SW1-1(config)# router ospf IPX
SW1-1(config-router)# int vlan 112
SW1-1(config-if)# ip router ospf IPX area 0
SW1-1(config-if)# int lo0
SW1-1(config-if)# ip router ospf IPX area 0
SW2
SW2(config-if)# feature ospf
SW2(config)# router ospf IPX
SW2(config-router)# int vlan 112
SW2(config-if)# ip router ospf IPX area 0
SW2(config-if)# int lo0
SW2(config-if)# ip router ospf IPX area 0
After configuration we can verify that switches can ping each other’s Loopback addresses.
Next up is the EIGRP configuration. In the question there is no mentioning about a EIGRP autonomous-
system number. So we can use any number we like in this case.
SW1-2
SW1-2(config-if)# feature eigrp
SW1-2(config)# router eigrp IPX
SW1-2(config-router)# autonomous-system 65000
SW1-2(config-router)# int vlan 123
SW1-2(config-if)# ip router eigrp IPX
SW1-2(config-if)# int lo0
SW1-2(config-if)# ip router eigrp IPX
SW3
SW3(config-if)# feature eigrp
SW3(config)# router eigrp IPX
SW3(config-router)# autonomous-system 65000
SW3(config-router)# int vlan 123
SW3(config-if)# ip router eigrp IPX
SW3(config-if)# int lo0
SW3(config-if)# ip router eigrp IPX
After configuration we can verify that switches can ping each other’s Loopback addresses.
SW3(config-if)#
SW3(config-if)# sh ip rout eigrp
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
The next question in this task is already a default configuration, so this doesn’t require any changes. It is
stated to make you aware that Graceful Restart or Non-Stop Forwarding is by default
configured on both EIGRP and OSPF configurations.
Next we configure timers in which we want to change the graceful-restart behavior. By default
the time-out of switches to reboot is set to 60 seconds. The question states that we will have a switch
connected to SW2 which takes longer to reboot. Therefore we need to change SW2 as that will be the
neighbor of the new switch.
SW2
SW2(config-if)# router ospf IPX
SW2(config-router)# graceful-restart grace-period 120
The next configuration is a bit more tricky. It comes down to the fact that you need to have
Graceful-Restart enabled on the EIGRP process before an In-Service-Software-Upgrade
will work. You will be warned when you disable the graceful-restart option and start an ISSU.
We can verify that it’s by default enabled and that we are capable of supporting ISSU’s!
In the EIGRP process we have a few different timers than are available in OSPF. In the EIGRP database
routes are maintained, but it’s already signaled to further EIGRP neighbors that one particular router in
the network is rebooting.
For this question we will change the hold-timer and the restart timer.
SW3
SW3(config-if)# router eigrp IPX
SW3(config-router)# timers nsf route-hold 300
SW3(config-router)# timers nsf signal ?
<10-360> Seconds
*Default value is 20
We should configure the fastest restart signaling timer. By using the question mark we find that the
timer can be set to a value between 10 and 360. This means that 10 seconds is the shortest time period
in which we can signal a restarting neighbor to other EIGRP neighbors.
Task 5: HSRP
This is the first of three tasks regarding a subject which is a lot alike. HSRP or Hot Standby
Routing Protocol is one of the 3 major First Hop Redundancy Protocols (FHRP).
These protocols ensure that for any given Layer 2 segment the Default Gateway is protected against
failures.
Clients can only set one default gateway IP address in their Network Interface Card (NIC) settings.
This gateway should therefore always be up, otherwise clients are disconnected from the network as
soon as 1 router fails.
The solution to this problem is a protocol which enables resiliency between routers while still being able
to have only 1 IP address as the default gateway for a Layer 2 segment (IP subnet).
This is achieved by using a Virtual IP address (VIP) which is configured on 2 switches. One of
the 2 switches will be the active or primary router and the other switch is the backup or hot-
standby. As soon as something happens with the primary switch, the next one will take over.
There are 3 major protocols that have similar functionality. The first that is going to be discussed is the
first protocol that was developed: HSRP.
The HSRP protocol is a Cisco proprietary FHRP. It uses a Virtual IP address connected to a Virtual MAC
address from the MAC address range of 0000.0C07.AC00 up to 0000.0C07.ACFF for HSRP
version 1 and 0000.0C9F.F000 up to 0000.0C9F.FFFF in HSRP version 2. Where the last
byte of the MAC address is related to the HSRP group number that is configured.
The primary HSRP router will be the only router that responds to ARP requests towards the Default
Gateway of the IP subnet. During the time both routers are active. The HSRP routers send Hello packets
to each other, informing them to be alive. Once a router fails and the hot-standby router reaches a
threshold of missing Hello packets it will become the new Primary router.
The selection of the primary router is done based on priority values that can be configured
(default 100). When the previously primary router comes back online, the HSRP protocol can, if
configured, fall back to the scenario where the highest priority router is the active router.
HSRP version 2 uses 224.0.0.102 as its Multicast group to signal to the other router. HSRP version 1
used the All-Routers multicast group of 224.0.0.2
HSRP version 2 uses a different virtual MAC address range, to ensure that there are no compatibility
errors between the 2 protocols. They are completely different in terms of communication
Under normal conditions in an HSRP environment there is always only 1 router active on any given time.
In vPC configurations the Nexus switches can do a trick where they can both be active for an HSRP
group. This is required, otherwise the vPC peer-link would be required to carry a lot of traffic between
the 2 switches as all incoming traffic on the non-primary vPC peer would need to send it’s traffic to the
primary vPC peer to be layer 3 switched.
In vPC environments both routers will respond to the HSRP Virtual MAC address. This is called ‘vPC
peer gateway’ which needs to be enabled. This requires that both vPC peers have an equal amount
of SVI or other routed interfaces to the Layer 3 network. Otherwise the vPC process can’t ensure a
successful failover. Without the peer-gateway command, the vPC switches will not respond to their
configured Virtual MAC Address, which may lead to a lot of traffic crossing the peer-link.
The initial configuration of this task is to configure the basic parameters of Cisco HSRP. We will be using
the 198.18.111.1 IP address as the default gateway for this segment. After configuring the basic
parameter (the Virtual IP address). The protocol already works!
SW1-1
SW1-1(config)# feature hsrp
SW1-1(config)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# ip 198.18.111.1
SW1-2
SW1-2(config)# feature hsrp
SW1-2(config)# int vlan 111
Copyright © by IPexpert. All rights reserved. 190
CCIE Data Center Lab Preparation Detailed Solution Guide
According to the MAC address tie-breaker we see that SW1-1 has become the Primary router for this
subnet and SW1-2 the hot-standby. For the group number we used the VLAN number. As there was no
indication in the question to use any specific group number.
The next question will decide which router will be the primary router in VLAN 111.
The question states that this should follow the best practice. If we take a look at the configuration guide
about this point. We see that Cisco is recommending to make the router HSRP primary if it also has
the vPC Primary role assigned. In our case it’s SW1-2 that is the primary vPC device in our network.
So we will make this the primary router in our HSRP network too.
There are no priority values given for this task so we can use anything we like. However, when you read
ahead of the questions. You will find a question related to Interface Tracking. These questions
ask you to subtract some value from the priority. If we use numbers now, it can help us later with these
questions.
We will use simple numbers for the priority values. I always keep as a best practice to make the standby
switch lower than the default value and the primary switch higher than the default value of the HSRP
priority, which is 100.
SW1-1
SW1-1(config)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# priority 90
SW1-2
SW1-2(config)# feature hsrp
Copyright © by IPexpert. All rights reserved. 191
CCIE Data Center Lab Preparation Detailed Solution Guide
After configuring the priority values and flapping the interface or the HSRP group. We see that SW1-2 is
now the active router.
Next is the security of the HSRP protocol. In the question it is asked to use an hashed key. Meaning that
we will be using MD5 authentication. This also immediately implies that we need to use HSRP
version 2, because we wouldn’t have the option to configure MD5 otherwise.
It’s a very dangerous configuration, because MD5 is perfectly configurable without HSRP version 2,
but the protocol keeps functioning without any authentication applied then!
Even show commands do not show you what is happening at that moment, but the traffic is not
authenticated without specifying the HSRP version 2.
In the question the authentication is also given a specific schedule. This means we will need to be using
a ‘key-chain’ to support this. Key chains are a mechanism to switch from authentication keys in a
given time schedule. This ensures security of the network and the exchange of keys. The timelines are
configurable and it’s also possible to accept an older key longer to ensure a smooth migration to the
new key.
SW1-1
SW1-1(config)# key chain HSRP
SW1-1(config-keychain)# key 1
SW1-1(config-keychain-key)# key-string IPexpertYEAR1
SW1-1(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec
31 2012
SW1-1(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan
01 2013
SW1-1(config-keychain-key)# key 2
SW1-1(config-keychain-key)# key-string IPexpertYEAR2
SW1-1(config-keychain-key)# send-life 00:00:00 Jan 01 2013 infinite
SW1-1(config-keychain-key)# accept-life 00:00:00 Jan 01 2013 infinite
SW1-1(config-keychain-key)# interface vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# authentication ?
WORD Plain text authentication string (Max Size 8)
md5 Use MD5 authentication
text Plain text authentication
The key chain configuration is done by using a number of keys. By using a send-lifetime and
accept-lifetime it’s configurable how long the key is sent and how long it is accepted as a valid
key. The initial key that is configured is sent from the current date and time up until December 31st. As
the question stated the key should be accepted for another 2 hours, to prevent time synchronization
issues between routers, which could cause an outage when authentication is rejected. The next (new)
key that is configured has an accept and send lifetime of infinite, because the question did not state
when the key should be changed again. Therefore we just use it forever.
The key chain is then configured on the HSRP configuration. Like previously mentioned, this is not
enough. HSRP will not use the MD5 configured authentication parameters when version 1 is used.
The HSRP version is not configured under the regular HSRP process configuration, but under the
interface itself. Version 2 is required for MD5 authentication. After this has been changed the HSRP
group uses the authentication parameters and authenticates the hello messages that are send and
received.
SW1-2
SW1-2(config)# key chain HSRP
SW1-2(config-keychain)# key 1
SW1-2(config-keychain-key)# key-string IPexpertYEAR1
SW1-2(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec
31 2012
SW1-2(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan
01 2013
SW1-2(config-keychain-key)# key 2
After both switches have been configured, the authentication is now used for the HSRP process.
SW1-2#
The following question relates to a feature called ‘pre-emption’. This feature ensures that when a
router comes online in an HSRP network and it has a higher priority configured it will assume the active
role. By default the current active router will keep its role when a higher priority router is seen. With
preempt enabled the router with the higher priority value will always take over (take back) it’s active
role.
The take over process can be delayed to ensure that the router that comes online (which is usually after
a reboot), is delayed before taking back it’s active role to have all other protocol running on this router
to synchronize correctly before it will attract traffic.
The question states that the active router has been rebooted and then comes online. This means that
the delay is configured for a reload to ensure all protocols have synchronized.
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
SW1-2
SW1-2(config-if-hsrp)# int vlan 111
SW1-2(config-if)# hsrp 111
SW1-2(config-if-hsrp)# preempt delay reload 180
SW1-2(config-if-hsrp)#
The next question is related to giving the HSRP process a name, which is an easy knob in the HSRP
configuration.
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# name IPexpertVLAN111
SW1-1(config-if-hsrp)#
SW1-2
SW1-2(config-if-hsrp)# int vlan 111
SW1-2(config-if)# hsrp 111
SW1-2(config-if-hsrp)# name IPexpertVLAN111
SW1-2(config-if-hsrp)#
Next are changing timers. With the default timers it’s only possible to configure up to a hello-
interval of 1 second. With the special knob ‘msec’ it’s possible to configure sub-second hello-
timers and therefore a convergence within 1 second
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# timers msec 300 msec 1000
SW1-1(config-if-hsrp)#
SW1-2
SW1-2(config-if-hsrp)# int vlan 111
SW1-2(config-if)# hsrp 111
SW1-2(config-if-hsrp)# timers msec 300 msec 1000
SW1-2(config-if-hsrp)#
The next 2 questions are related to HSRP object tracking. This tracking ensures that when the tracking
group fails, the HSRP priority is lowered. In legacy scenarios when the priority falls under the priority of
the hot-standby router and pre-emption is configured. The hot-standby takes over the active role.
In this scenario on NX-OS with vPC’s on the switch. The failure of a switch will result in a failover to the
hot-standby router of course. But there is no single distinction anymore between the active and hot-
standby router. Both routers are forwarding traffic. What can be configured by using the priority
command is to configure thresholds when the traffic forwarding should be stopped and traffic is no
longer layer 3 routed on the NX-OS device, but rather send across the vPC Peer-link towards the vPC
peer for layer 3 routing.
In our case of the questions, it should take 2 uplinks to fail on the active switch before it stops
forwarding traffic on Layer 3 and sending traffic on a Layer 2 basis towards the vPC peer.
Currently we have a priority of 110 configured on the primary switch (or any other number that you
configured in a previous question). 10% of this value should be subtracted once an uplink fails. When a
second interface fails it should stop layer 3 forwarding completely.
10% of the 110 value is 11. So we subtract 11 from each of the failing Ethernet uplinks. Then we
configure the thresholds for forwarding appropriately. Remember to use a different value for the hot-
standby switch as 10% from 90 is only 9 and not 11.
Depending on the values you chose at the earlier question. You need to ensure that when you subtract 2
times 10% from that value, it should be lower than the hot-standby switch, so it will assume the active
role after 2 uplinks have failed.
In our case the priority value will drop with 22 (2 times 11) and therefore be 88. Therefore the hot-
standby switch will take over as it’s configured for priority 90.
Remember that pre-emption needs to be configured for a fail-over when priority drops
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# priority 90 forwarding-threshold lower 72 upper 81
SW1-1(config-if-hsrp)# exit
SW1-1(config-if)# exit
SW1-1(config)# track 1 ?
<CR>
interface Interface to track
ip IPv4 protocol
ipv6 IPv6 protocol
list Object tracking list
SW1-1(config-track)# exit
SW1-1(config)# int vlan 111
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# track 1 decrement 9
SW1-1(config-if-hsrp)# track 2 decrement 9
SW1-1(config-if-hsrp)#
SW1-2
SW1-2(config)# track 1 interface ethernet3/7 line-protocol
SW1-2(config-track)# exit
SW1-2(config)# track 2 interface e3/9 line-protocol
SW1-2(config-track)# exit
SW1-2(config)# int vlan 111
SW1-2(config-if)# hsrp 111
SW1-2(config-if-hsrp)# priority 110 forwarding-threshold lower 88 upper 99
SW1-2(config-if-hsrp)# track 1 decrement 11
SW1-2(config-if-hsrp)# track 2 decrement 11
Interface tracking can be done using multiple configuration options. The question now states that we
need to track an interface for going down. This means we do not need to do anything with the IP related
tracking features for now. We can just track on line-protocol going down and the tracking group will go
down.
The final question of this task is to use a physical interface MAC address instead of a virtual
MAC. Some systems might have trouble seeing special MAC addresses like the one used by HSRP.
Therefore there is a fallback to a physical MAC address to use as Virtual. This feature will ‘borrow’ a
MAC address from the physical interface or SVI from the Active router. When the active router fails it’s
MAC address is then moved just like a regular HSRP action to the hot-standby router. The MAC address
will stay the same even though it does not belong to the new active router.
The configuration is fairly easy, although not specified under the HSRP configuration, but under the
interface directly.
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
SW1-1(config-if)# hsrp use-bia
SW1-2
SW1-2(config-if-hsrp)# int vlan 111
SW1-2(config-if)# hsrp use-bia
If we now check the HSRP group, we see that it’s using a real MAC address as it’s virtual MAC address.
SW1-1(config-if)# sh hsrp
Vlan111 - Group 111 (HSRP-V2) (IPv4)
Local state is Active, priority 72 (Cfged 90)
Forwarding threshold(for vPC), lower: 72 upper: 81
SW1-1(config-if)#
Next task is about the standards based solution based on the Cisco proprietary HSRP protocol!
Task 6: VRRP
The second FHRP protocol that is discussed is the Virtual Router Redundancy Protocol (VRRP). This
protocol is very similar in operation to Cisco’s HSRP. It borrows a lot of technical specifications. This
protocol is an open standard based on RFC 3768 (version 2) or RFC 5798 (version 3).
All routing vendors have implementations for the VRRP. This makes it the best practice protocol to use
when you are using a mixed-vendor environment. Cisco is not active in implementing new features in
VRRP compared to HSRP. A lot of advanced features like MD5 authentication or IPv6 support are not
available in latest releases of NX-OS. Version 3 of VRRP (which brings IPv6 support) is not available in
NX-OS yet. Other features like BFD tracking, which have long been available in HSRP, are only recently
implemented for VRRP.
The questions in this task are very similar to the questions in the HSRP task as the protocol works almost
the same as HSRP.
The first few questions are related to configuring the basic configuration parameters of the VRRP. The
third question however is asking for a special configuration. A special virtual MAC address should be
used for the VRRP group.
VRRP uses MAC address from the range of 0000.5E00.0100 to 0000.5E00.01FF. We are asked
to configure a virtual MAC address of 0000.5E00.0174. This means we should use a special group
number. As the group number is the last byte of the MAC address, just like with HSRP.
When you convert the number 74 from hexadecimal to decimal we do the following steps:
0111
0100
A hex character consists of 2 nibbles making up 1 byte. We put the 2 nibbles together and convert the
binary byte to a decimal number:
01110100 = 116
SW1-1
SW1-1(config-if)# int vlan 121
SW1-1(config-if)# vrrp 116
SW1-1(config-if-vrrp)# address 198.18.121.254
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# address 198.18.121.254
SW1-2(config-if-vrrp)# priority 200
After configuration, it seems that the VRRP group is not working correctly.
SW1-2(config-if)# sh vrrp
Interface VR IpVersion Pri Time Pre State VR IP addr
---------------------------------------------------------------
Vlan121 116 IPV4 200 1 s Y Init 198.18.121.254
This is a very strange message as the reason why it’s staying on “Init” is very clear when you take a
closer look.
It’s clearly seen that the VRRP group is by default on a shutdown state and it needs to be enabled
before it will start sending VRRP packets.
We can immediately see that the virtual MAC address is as it should be per the question. This means we
did our calculation correctly.
SW1-1
SW1-1(config-if)# int vlan 121
SW1-1(config-if)# vrrp 116
SW1-1(config-if-vrrp)# no shutdown
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# no shutdown
SW1-2(config-if-vrrp)# sh vrrp
Interface VR IpVersion Pri Time Pre State VR IP addr
---------------------------------------------------------------
Vlan121 116 IPV4 200 1 s Y Master 198.18.121.254
The next task asks to configure authentication on the VRRP group. MD5 authentication is not supported
in NX-OS at the time of writing. Authentication is therefore easily configured.
SW1-1
SW1-1(config-if)# int vlan 121
SW1-1(config-if)# vrrp 116
SW1-1(config-if-vrrp)# authentication text IPexpert
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# authentication text IPexpert
Now we verify we indeed see that the process is authenticated on the routers
One thing different from the Cisco HSRP is that pre-emption is enabled by default. Again this ensures
that when a higher priority router comes online on the subnet, the routers will take over the active role
pro-actively. In the question is stated that we don’t want this to happen, so we should disable it.
We are asked to only configure the Active switch. This means that an initial failover will happen right
away due to pre-emption, but a fall-back will only occur when the other switch fails.
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# no preempt
The next question is a little more tricky. VRRP does not support the tracking of Layer 2 interfaces. It does
support direct tracking of Layer 3 interfaces, where no tracking groups are necessary. Instead of tracking
an interface in this question we are tracking the reachability of a route in the routing table.
This means that as soon as SW3 fails, the Loopback address will disappear from the routing table on
SW1-2. This as per our question means we need to do a failover to the other VRRP router. This process
should be delayed with 30 seconds. Remember to not override your previous tracking group
configuration, otherwise you will lose points in the HSRP task!
SW1-2
SW1-2(config)# track 3 ip route 198.18.0.3/32 reachability
SW1-2(config-track)# ?
delay Tracking delay
no Negate a command or set its defaults
vrf Configure VPN Routing/Forwarding table
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1-2(config-track)# delay ?
down Delay down change notification
up Delay up change notification
There is no specification how much the group should be decremented. Therefore we just ensure that
the priority value is decreased under the default of 100. This would cause the switch to take over the
active role as we did not disable pre-emption on the other switch.
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# track 1 decrement 101
The final question of this task is to configure timers of VRRP. The only trick you have to change timers is
to change the interval in which VRRP packets are advertised to all VRRP running routers on the
subnet. The holddown timer is not adjustable. This is by default 3 times the hello-interval including
some priority value timing. For any given configured priority value setting the timers are adjusted a little
bit, which is the way how VRRP elects it’s active and standby router.
If we want to reach a down signaling within 10 seconds, we will use a hello-interval of 3 seconds. When
3 hello packets are missed after 9 seconds including a little priority timing we will signal a VRRP
neighbor down within 10 seconds.
SW1-1
SW1-1(config-if)# int vlan 121
SW1-1(config-if)# vrrp 116
SW1-1(config-if-vrrp)# advertisement-interval 3
SW1-2
SW1-2(config-if)# int vlan 121
SW1-2(config-if)# vrrp 116
SW1-2(config-if-vrrp)# advertisement-interval 3
Now the VRRP task is finished. The final task for FHRP protocols is another Cisco proprietary protocol
supporting Load Balancing.
Task 7: GLBP
The last but not least FHRP that is tested in our workbook is another Cisco proprietary protocol called
the Gateway LoadBalancing Protocol (GLBP). This protocol is the most recent one of all
FHRP’s. It uses a little different approach than HSRP and VRRP. The other FHRP’s are using a
single active router in normal environments. In vPC environments this is a little different as
both HSRP and VRRP gateways are active and responding to client requests. The GLBP protocol
supports this by default, but it’s also possible to configure the load balancing , which is not possible in
case of the vPC load balancing trick.
These are routers that are forwarding Layer 3 traffic according to the weight that is configured on them
to allow the traffic coming in.
AVFs have their own dedicated Virtual MAC Address for each separate AVF
This is only a single router in the GLBP configuration which is selected based on priority values.
The Active Virtual Gateway is aware of all Virtual MAC Addresses in the GLBP configuration and uses a
Round Robin, Weighted or Host Dependent mechanism to answer the ARP requests. This
means that each ARP request can have a different MAC address in the ARP reply. This is how the load
balancing is achieved.
The failover of an AVF is handled using 2 timers. The Redirect Timer is used to gracefully take out
the Virtual MAC address of the AVF of the Virtual MAC Pool maintained by the AVG. The Secondary
Hold Timer is used when a AVF has completely failed and the Virtual MAC Address of this AVF will be
freed in the pool and is available for possible new AVF routers.
There are 3 ways of answering the ARP requests on the AVG and achieving load balancing. These are
explained in the following table.
Mode Description
Round Robin This is the most granular way of load balancing as the AVG answers a other
Virtual MAC Address on each ARP request. Meaning traffic is equally load
balanced across all AVF’s
Weighted This mode enables the use of weight values on AVFs. When a higher weight
is advertised by an AVF. The AVG will answer more ARP requests with this
Virtual MAC Address so this AVF will handle more traffic than others
Host dependent The host dependent mode guarantees that the same host receives the
same AVF each time it does an ARP request.
You can change the weight or priority values for becoming an AVF or AVG according to interface
tracking, just like in the other FHRPs.
In the current software releases used for this version of the CCIE Datacenter test the GLBP is only
supported on the Nexus 7000 series switches. Additionally there is currently no support for IPv6 in the
protocol.
Now to start our configuration we begin with fairly basic tasks of configuring the GLBP on VLAN 222
on our Nexus 7000 series switches.
SW1-1
SW1-1(config-if)# feature glbp
SW1-1(config)# int vlan 222
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# ip 198.18.222.55
SW1-1(config-if-glbp)# no shutdown
SW1-2
SW1-2(config-if)# feature glbp
SW1-2(config)# int vlan 222
SW1-2(config-if)# glbp 222
SW1-2(config-if-glbp)# ip 198.18.222.55
SW1-2(config-if-glbp)# no shutdown
The next 2 tasks first require to assign the AVG role to SW1-1 and that both routers should be AVF. By
default all GLBP capable routers are used for load balancing traffic to. A router can decide for itself if it
does not want to participate in the load balancing anymore, by configuring upper and lower limits. We
will get there in a later question in this task.
SW1-1
SW1-1(config)# int vlan 222
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# priority 110
The next question is related to the previous one. We need to ensure that both routers are having an AVF
role in the network, but do want to remove the router as an AVF when the Loopback address of the
upstream switches disappears from the routing table.
Just like in the VRRP task we can use IP Route Reachability tracking for this. We will configure
lower thresholds when the router will give up its AVF role. Again do not forget to use unique numbers
for the tracking groups. Otherwise you will overwrite configuration from a previous task.
SW1-1
SW1-1(config)# track 4 ip route 198.18.0.2/32 reachability
SW1-1(config)# int vlan 222
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# track 1 decrement 20
SW1-1(config-if-glbp)# weighting 110 lower 90 upper 110
SW1-2
SW1-2(config)# track 4 ip route 198.18.0.3/32 reachability
SW1-2(config)# int vlan 222
SW1-2(config-if)# glbp 222
SW1-2(config-if-glbp)# track 1 decrement 20
SW1-2(config-if-glbp)# weighting 110 lower 95 upper 105
With this configuration we will take out the router from the AVF role (and the AVG from its role when
applicable) when the reachability towards a IP prefix is gone.
This process should be delayed. We can do this in 2 ways. The first is what we did in the VRRP task, by
delaying the tracking group for signaling the loss of reachability. If you read the question carefully it
mentions that you need to support yourself for a failure of a current AVF. The AVG will only take out a
failed AVF from the pool after the redirect timer has expired, after which it will keep its Virtual
MAC reserved for another timer (secondary timeout timer).
When we change the redirect timer the AVG will take out the AVF and that is what this question is
asking for. Keep in mind to change both switches as they both might become the AVG at some point.
SW1-1
SW1-1(config)# int vlan 222
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# timers redirect 180
SW1-2
SW1-2(config)# int vlan 222
SW1-2(config-if)# glbp 222
SW1-2(config-if-glbp)# timers redirect 180
The next task is regarding the pre-emption of the AVG role. The preempt command is disabled by
default in the GLBP protocol. By default the delay for taking over from a lower priority router is set to
3600 seconds. This is significantly longer than any other timer in a FHRP. The reason for this is that a
router immediately becomes an AVF when it comes online, so it’s already forwarding traffic. The
switchover of the AVG role is therefore not that important anymore.
SW1-1
SW1-1(config)# int vlan 222
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# preempt delay minimum 30
SW1-2
SW1-2(config)# int vlan 222
SW1-2(config-if)# glbp 222
SW1-2(config-if-glbp)# preempt delay minimum 30
The final task is regarding ISSU’s on GLBP enabled routers. The GLBP protocol supports a nice feature for
supporting Non Stop Forwarding. You can set timers as short as you like, but when a restart is
done of any neighbor or the router itself, it can temporarily extend the timers for the duration of
the Graceful Restart.
SW1-1
SW1-1(config)# glbp timers extended-hold
SW1-2
SW1-2(config)# glbp timers extended-hold
The FabricPath network uses Equal Cost MultiPathing to reach any given destination (MAC
address) in the network. Now when any FabricPath enabled router will perform a lookup on the next-
hop it will result in a Destination Switch ID. For vPC connected hosts or switches this is
different as they are connected to 2 different FabricPath Switch IDs. It will result in strange
behavior and non-deterministic traffic flows throughout the FabricPath network.
The drawing illustrates that each packet, depending on the Port-channel hashing can be sent from or to
either S1 or S2. Everytime S3 learns the MAC address of the PC connected it will update its routing
table accordingly. This introduces a MAC address flapping issue on S3.
The solution that Cisco came up with has been called vPC+. This means that the vPC switches are
compatible with FabricPath enabled networks.
The solution is that the whole vPC domain will be virtualized as a single FabricPath Switch ID. In that way
the entry for the MAC address of the PC connected has a Destination Switch ID which is the virtual ID
configured statically on the vPC domain. That way the other FabricPath switches can still use Equal Cost
Multi-Pathing towards the host just like any other destination in the FabricPath network without causing
any loops.
The configuration is fairly simple. The task asks you to configure the FabricPath network first after which
it should be made ready for the vPC configuration.
As there is no host connected to the switches, you just need to prepare the configuration for it. After
this pre-configuration there is nothing different from a standard vPC configuration.
The topology that will be used for this lab is laid out in Drawing 3 in the workbook and for your
convenience showed below:
SW2
SW2(config)#
SW2(config)# feature-set fabricpath
SW2(config)# vlan 666
SW2(config-vlan)# mode fabricpath
SW2(config)# interface Ethernet1/1
SW2(config-if)# switchport mode fabricpath
SW2(config)# interface Ethernet1/5
SW2(config-if)# switchport mode fabricpath
SW3
SW3(config)#
SW3(config)# feature-set fabricpath
SW3(config)# vlan 666
SW3(config-vlan)# mode fabricpath
SW3(config)# interface Ethernet1/1
SW3(config-if)# switchport mode fabricpath
SW3(config)# interface Ethernet1/5
SW3(config-if)# switchport mode fabricpath
SW1-1
SW1-1(config)#
SW1-1(config)# feature-set fabricpath
SW1-1(config)# vlan 666
SW1-1(config-vlan)# mode fabricpath
SW1-1(config)# interface Ethernet2/9
SW1-1(config-if)# switchport mode fabricpath
SW1-1(config)# interface Ethernet2/11
SW1-1(config-if)# switchport mode fabricpath
SW1-2
SW1-2(config)#
SW1-2(config)# feature-set fabricpath
SW1-2(config)# vlan 666
SW1-2(config-vlan)# mode fabricpath
SW1-2(config)# interface Ethernet2/13
SW1-2(config-if)# switchport mode fabricpath
SW1-2(config)# interface Ethernet2/15
SW1-2(config-if)# switchport mode fabricpath
Now to finish the chapter we need to configure the static FabricPath Switch ID under the vPC domain
configuration.
SW2
SW2(config)#
SW2(config)# vpc domain 100
SW2(config-vpc-domain)# fabricpath switch-id 100
SW3
SW3(config)#
SW3(config)# vpc domain 100
SW3(config-vpc-domain)# fabricpath switch-id 100
You finished Chapter 4 successfully! We also finished all Networking chapters at this moment and we
will continue with the Fibre-Channel features of both the Cisco MDS switches and the Nexus 5000
switches.
Chapter 5: Data
Center Storage
Networking
Chapter 5: Data Center Storage networking is intended to let you be familiar with the Storage
Networking features on the Cisco MDS switches. Configuring traditional Fibre Channel networks and
basic Fibre Channel features.
We highly recommend creating your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab.
General Rules
Try to diagram out the task. Draw your own connections the way you like it
Take a very close read of the tasks to ensure you don’t miss any points during grading!
Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular
chapter
Solutions
However, the Fibre Channel world is completely different from the Ethernet world and therefore we
require some extensive configuration to have a good understanding of the Fibre Channel protocol and
the many differences that there are with Ethernet switching.
The Fibre Channel protocol has been around for a long time already and is used widely in the world. It
makes it possible for servers to extend the SCSI commands to hard disks over a data network. This is the
initial purpose of the Fibre Channel network. Fibre Channel works a little different in the OSI model than
Ethernet works.
You can say that the Fibre Channel protocol is much more intelligent than Ethernet. Most of the routing
protocols that we need to implement in the Ethernet and IP world are by default enabled in the Fibre
Channel world and do not require any configuration to work properly. Most of the protocols can come
online by default and are automatically converging fine.
The Fibre Channel protocol uses a special ‘ID’ of disks and hosts (or targets and initiators) that identify
the host towards the network. These ID’s are called World Wide Names (WWNs). A device always has
2 of these ID’s, these are a World Wide Port Name (WWPN) and a World Wide Node Name
(WWNN). It makes sense that the WWPN is assigned for every port on the device and the WWNN is assigned
to a single device. A special WWN is assigned to every MDS switch. These are called Switch World
Wide Names (SWWNs). These ID’s are used to identify any MDS switch in a topology and can be used
to reference a switch. The ID is also used by the switch in many of the control-plane protocols that are
running in the FC control-plane.
These IDs are most comparable to MAC addresses in the Ethernet world, except that the WWNs are
never used in the data communication! This is a common misconception that, like in Ethernet, the
physical addresses are used as Layer 2 protocol. They are just for identifying the host or port towards
the Fibre Channel environment.
After the identification the FC switch will hand out the logical addressing (like IP addresses in the
Ethernet/IP world). These addresses are called FCIDs. They are comprised of a 3-byte / 24-bit
number represented as 6 hexadecimal characters: 0x890ABC. The first byte of the FCID is the so-called
FC Domain ID of the MDS switch. The second byte is the area code and the third byte is the
address of the host. The Domain ID is an ID used by the Fibre Channel environment between FC
switches for routing!
The first byte of an FCID is used and a routing lookup is done against it to determine the destination FC
switch in the network. When the destination switch is reached, the area code is used to determine
the exit interface of this packet. Usually in an FC environment a whole area code is handed out to one
interface on the switch. After that each host/disk that is connected behind that interface, in case it’s an
NPV port or a Loop port, different devices get an ID from that area handed out.
The model looks a bit like a DHCP environment, except it’s distributed and each FC switch is participating
in this handing out of addresses. There is one switch in the network responsible for handing out
Domain IDs to FC switches and this is the Principal switch. This Principal switch is chosen
according to the highest priority number of as a tie-breaker the highest SWWN.
This model is supported by some control-plane protocols that each host on the network needs to
perform when it is connected to a switch. This process is called an FLOGI or Fabric Login process.
After the FLOGI a Port Login or PLOGI is done to identify all capabilities of the just connected
host.
In the configuration tasks more is explained about the Fibre Channel protocol.
The topology we use in this lab is simple. We have 2 native MDS switches. In a later chapter we will
include the Nexus switches in the Fibre Channel topology as well, but we are currently only focusing on
the native Fibre Channel topologies.
Connected to the 2 MDS switches are 2 JBODS. These devices are nothing more than containers for SCSI
disks that are capable of logging in to a Fibre Channel switch. Fibre Channel attached servers are then
able to connect to this physical hard-disk through the Fibre Channel network.
In the real world these disks would communicate to a Storage Filer where RAID configurations and
other access methods are applied.
Our topology will use the disks as dual-attached devices where we can simulate having a lot of devices
on the Fibre Channel network.
During this chapter the detailed solution guide will explain Fibre Channel (FC) concepts. This will not be
an in-depth explanation, but a basic explanation to ensure you understand the concept and how it’s
related to our CCIE Data Center topology.
The MDS switches will boot up with an empty configuration where we need to apply all configuration.
The MDS switches run the same code as the Nexus switches. The old SAN-OS has been rebranded to
NX-OS since version 4.0. The CLI is exactly the same, except that the FC protocol all has their own
configuration parameters which are not compatible with their Ethernet equivalents.
Initially we need to give those hostnames and configure the mgmt0 management Ethernet interfaces.
MDS1
Invalid admin password. Please try again.
This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Please register Cisco MDS 9000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. MDS devices must be registered to receive entitled
support services.
MDS Switch
10.2.8.74 login: admin
Password: IPexpert123
Cisco Storage Area Networking Operating System (SAN-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
switch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# switchname MDS1
MDS1(config)# interface mgmt0
MDS1(config-if)# ip address 172.16.100.10/24
MDS1(config-if)# no shutdown
MDS1(config-if)# ip default-gateway 172.16.100.254
MDS1(config)# ntp server 172.16.100.254
MDS2
switch(config)#
switch(config)# switchname MDS2
MDS2(config)# interface mgmt0
MDS2(config-if)# ip address 172.16.100.11/24
MDS2(config-if)# no shutdown
MDS2(config-if)# ip default-gateway 172.16.100.254
MDS2(config)# ntp server 172.16.100.254
Verify that time is synchronized. Keep in mind that this might take up to 5 minutes before a full
synchronization is completed.
The lab states that we shouldn’t use any automatic selection for interface speeds. This means that we
need to specify static link speeds on every link that connects a host to our MDS switches. Besides that
we shouldn’t use a feature of the MDS switches that let’s the switch automatically select the type of
interface that should be selected according to the connected device.
The type of interface is a crucial point in configuring interfaces in a Fibre Channel network. This changes
the behavior of the interface and the FC protocol. The Cisco MDS switches have a feature that they can
auto detect what kind of interface is connected on the other side and set the interface type according to
that.
The following table shows all the various interface types, which are configurable on the MDS switch and
what they do:
Mode Description
E The E-port is a so-called Fabric Extension port. This port ensures the extension of the
Fibre Channel fabric. A connection between 2 switches is always configured as an E-
port. Pay attention to oversubscription on E-ports. You might need more buffer
credits on these ports as a lot of traffic might go across them.
TE The TE-port is equal to the E-port in positioning, except that this port has the
Trunking protocol enabled, which enables the use of VSANs between Cisco MDS
switches.
F The Fabric Port is the most common interface. You can attach Hosts and JBODs to
these interfaces. On the Host or JBOD itself it’s called an N-port. So an N-port always
connects to an F-port.
FL A Fabric Loop port is used to connect Public Arbitrated Loops to the MDS switch.
Arbitrated Loops are self-supporting FC rings without switches between devices. This
is a way to connect them to an FC infrastructure
Fx The Fx port can be either an F or an FL port. It will automatically detect if it’s a normal
N-port connected behind of it or an Arbitrated Loop. This mode can be the default
mode on switches.
TF The TF port is equal to an F-port, but supports the Trunking protocol, to connect
VSANs towards hosts or NPV switches
NP NP ports are equal to N-ports on a device, but these ports are on FC switches that are
set to N-Port Virtualization mode.
TNP These ports are equal to NP ports, but support the Trunking (EISL) protocol, to
transfer VSAN information to NPV switches
SD SD is the SPAN Destination port where a FC analyzer can be connected, like the Cisco
FC to Ethernet converter. This traffic receives traffic from other ports for analyzation
ST SPAN Tunnel ports are used as ‘reflector’ ports for Remote SPAN sessions. Traffic that
needs to be copied to another device is using this tunnel port for encapsulation.
Mode Description
TL This port connects Translative Loops to the MDS switches. It enables communication
between TL, FL and other devices in the FC fabric
Auto This is the default mode on most MDS switches. This mode auto detects the mode in
which it needs to operate in. It can become a E, F, FL, TE or TF port. Other port modes
need to be manually configured.
In this exercise we shouldn’t use the automatic interface type selection so we need to configure it
statically. This is so you practice with the various port types that we need.
In the CCIE Data Center lab it’s typical that you will be stated what kind of JBODs are attached to the
switches. These can be anything, from an N port to any kind of Private or Public loop. Be very careful as
you need to select the correct interface type, because when this is not properly configured you will not
see any FLOGI’s from the attached devices.
There are a few more requirements that we need to comply with. We need to use 200MBps
connections towards the JBODs. Pay attention to the capital B in this word. Fibre Channel can be
configured to work in 1Gbps, 2Gbps, 4Gbps or 8Gbps speeds. However, the protocol uses an
8B/10B encoding mechanism, which means that for every 8 bits of data, 10 bits are transferred over
the link. This comes down to a perfect match in throughput that 1Gbps FC traffic is 100MBps of data
traffic and so on.
Now we start configuring the interfaces. You might need to detect how the JBODs are connected. Just
try a few interface types until you see FLOGIs coming in.
In our example the JBODs are using Public Arbitrated Loops which means we need FL ports.
The ISLs between the switches should be using a standard based solution. Which means they should use
a normal ISL protocol and not the Cisco specific (although also standardized now, nobody implements it)
EISL encapsulation. We can use the maximum interface bandwidth, which is 4Gbps on our interfaces
between the switches.
Pay attention that you need to use a dedicated rate-mode on certain switches for configuring E -ports.
This has to do with the way that the ASIC is used. On E-ports all B2B credits are assigned to this interface
as it’s
On E-ports all B2B credits located on the ASIC are assigned to this interface as it’s required for high
bandwidth and longer links.
Finally, all interfaces on the MDS switches are by default set to shutdown (unless specifically
configured).
MDS1
MDS1(config)# int fc1/5
MDS1(config-if)# sw mode fl
MDS1(config-if)# sw speed 2000
MDS1(config-if)# no shut
MDS1(config-if)# int fc1/6
MDS1(config-if)# sw mode fl
MDS1(config-if)# sw speed 2000
MDS1(config-if)# no shut
MDS1(config-if)# int fc1/1
MDS1(config-if)# sw mode e
MDS1(config-if)# sw trunk mode off
MDS1(config-if)# sw speed 4000
MDS1(config-if)# no shut
MDS1(config-if)# int fc1/2
MDS1(config-if)# sw mode e
MDS1(config-if)# sw trunk mode off
MDS1(config-if)# sw speed 4000
MDS1(config-if)# no shut
MDS1(config-if)# int fc1/3
MDS1(config-if)# sw mode e
MDS1(config-if)# sw trunk mode off
MDS1(config-if)# sw speed 4000
MDS1(config-if)# no shut
MDS1(config-if)# int fc1/4
MDS1(config-if)# sw mode e
MDS1(config-if)# sw trunk mode off
MDS1(config-if)# sw speed 4000
MDS1(config-if)# no shut
Keep in mind that by default the EISL protocol is enabled on E ports, so ports between MDS switches
automatically become TE ports, when nothing is changed.
MDS1(config-if)#
We are now using standard based ISLs and we configured a few JBOD’s to the MDS switches. If
everything is working correctly. We should see FLOGI’s being done to the switch:
MDS1(config-if)#
And it’s working! We see disks from the JBOD logging in to the MDS switch successfully!
Now we need to configure the second switch. The task here is that the JBOD interfaces should use
automatic selection of speeds, but still should comply with the previous task of 200MBps transfer rates.
This is done by specifying a maximum speed of the automatic selection.
MDS2
MDS2(config)# int fc1/5
MDS2(config-if)# sw mode fl
MDS2(config-if)# sw speed auto max 2000
MDS2(config-if)# no shut
MDS2(config-if)# int fc1/6
MDS2(config-if)# sw mode fl
MDS2(config-if)# sw speed auto max 2000
MDS2(config-if)# no shut
MDS2(config-if)# int fc1/1
MDS2(config-if)# sw mode e
MDS2(config-if)# sw trunk mode off
MDS2(config-if)# sw speed 4000
MDS2(config-if)# no shut
MDS2(config-if)# int fc1/2
MDS2(config-if)# sw mode e
MDS2(config-if)# sw trunk mode off
MDS2(config-if)# sw speed 4000
MDS2(config-if)# no shut
MDS2(config-if)# int fc1/3
MDS2(config-if)# sw mode e
MDS2(config-if)# sw trunk mode off
MDS2(config-if)# sw speed 4000
MDS2(config-if)# no shut
MDS2(config-if)# int fc1/4
MDS2(config-if)# sw mode e
MDS1
MDS1(config)# int fc1/5
MDS1(config-if)# sw owner JBOD1A
MDS1(config-if)# int fc1/6
MDS1(config-if)# sw owner JBOD2A
MDS1(config-if)# int fc1/1
MDS1(config-if)# sw owner MDS2 port 1
MDS1(config-if)# int fc1/2
MDS1(config-if)# sw owner MDS2 port 2
MDS1(config-if)# int fc1/3
MDS1(config-if)# sw owner MDS2 port 3
MDS1(config-if)# int fc1/4
MDS1(config-if)# sw owner MDS2 port 4
MDS2
MDS2(config)# int fc1/5
MDS2(config-if)# sw owner JBOD1B
MDS2(config-if)# int fc1/6
MDS2(config-if)# sw owner JBOD2B
MDS2(config-if)# int fc1/1
MDS2(config-if)# sw owner MDS1 port 1
MDS2(config-if)# int fc1/2
MDS2(config-if)# sw owner MDS1 port 2
MDS2(config-if)# int fc1/3
MDS2(config-if)# sw owner MDS1 port 3
MDS2(config-if)# int fc1/4
MDS2(config-if)# sw owner MDS1 port 4
One of the ports on MDS1 should be easily located. This is done using a blinking switchport LED known
as the ‘beacon’. This is a single command which enables you to physically find interfaces in high port
density switches.
MDS1
MDS1(config)# int fc1/5
MDS1(config-if)# sw beacon
The next feature is a feature used for long distance links. Those links are typically using lot’s of
interconnects and it could occur that errors are risen during the transport of a frame across this fiber.
The MDS switches use certain thresholds for these errors and after a threshold is reached the interface
is put in err-disabled mode because of this.
The thresholds are reached when 15 error bursts occur in a 5 minute period. The errors might occur
because of one of the following reasons:
• Bad fiber
• Bad SFP
• Short haul cable is used for long haul or long haul cable is used for short haul
• We can disable this if we know the link is fine and we want to overrule the thresholds. As there
is no MDS specified in this task we will configure both.
MDS1
MDS1(config)# int fc1/10
MDS1(config-if)# sw ignore bit-errors
MDS2
MDS2(config)# int fc1/10
MDS2(config-if)# sw ignore bit-errors
Next we need to ensure that all interfaces are shutdown without any configuration. This is where we are
configuring defaults.
MDS1
MDS1(config)# system default ?
switchport Configure default values for switchport attributes
zone Configure default values for zone
MDS1(config)# system default switchport ?
mode Enter the port mode
shutdown Disable/enable switchports by default
trunk Configure trunking parameters as a default
MDS1(config)# system default switchport shutdown
MDS1(config)#
MDS2
MDS2(config)# system default switchport shutdown
MDS2(config)#
The next task is regarding giving specific devices within the Fibre Channel an easily remembered name
or alias. This alias should be known throughout the fabric. We can do this using 2 ways. One of
them is by using a standard based solution called ‘FCalias’. This is a method of giving names to
certain characteristics of a connected host (WWPN, Interface, FCID, etc.). The only problem is that
FCalias is dependent on the VSAN. You cannot create a VSAN overlapping FCalias. Besides that it’s
not possible to distribute the FCalias information throughout the Fibre Channel fabric.
The solution we are looking for here is a Cisco proprietary solution called device-alias. These
device-aliases can be distributed throughout the FC Fabric by using the Cisco Fabric Services (CFS). You
can create a device-alias using a PWWN of the device. By default when you the use this device-alias
in any kind of configuration, for example Zoning, it will automatically convert back to the PWWN of the
device.
What you really want is that the device-alias remains the connection between the zone and the
host, so that when changes are done to the device-alias these are leveraged back into the zoning
database automatically. This is done using so-called Enhanced device-alias. Then the device-
alias name is kept in the zoning configuration, just like our task asks for.
Do not forget to enable the distribution of device-aliases, because this is not enabled by default.
You can use the FLOGI database for filling the device-alias database.
MDS1
MDS1(config)# device-alias distribute
MDS1(config)# device-alias mode enhanced
MDS1(config)# device-alias database
MDS1(config-device-alias-db)# device-alias name J1D1 pwwn
c6:83:00:11:0d:18:fa:ce
MDS1(config-device-alias-db)# device-alias name J1D2 pwwn
c6:83:00:11:0d:18:fb:ce
MDS1(config-device-alias-db)# device-alias name J1D3 pwwn
c6:83:00:11:0d:18:fc:ce
MDS1(config-device-alias-db)# device-alias name J1D4 pwwn
c6:83:00:11:0d:18:fd:ce
MDS1(config-device-alias-db)# device-alias name J2D1 pwwn
21:00:00:11:0d:18:fe:ce
MDS1(config-device-alias-db)# device-alias name J2D2 pwwn
21:00:00:11:0d:18:ff:ce
MDS1(config-device-alias-db)# device-alias name J2D3 pwwn
21:00:00:11:0d:18:a0:ce
MDS1(config-device-alias-db)# device-alias name J2D4 pwwn
21:00:00:11:0d:18:a1:ce
MDS1(config-device-alias-db)# device-alias commit
Now we need to verify if we see all the aliases on the other MDS as well, without configuring it. Be
aware that your ISLs are up before this works!
MDS1
MDS1(config)# do sh device-alias database
device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce
device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce
device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce
device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce
device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce
device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce
device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce
device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce
MDS2
MDS2(config)# do sh device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 8 Mode: Enhanced
Checksum: 0x617152b263ae5b4ebb547a6fb633b7fb
MDS2(config)#
MDS2(config)# do sh device-alias database
device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce
device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce
device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce
device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce
device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce
device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce
device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce
device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce
B2B credit buffers are assigned to an interface. The buffers are used for the fact of the no-drop policy
that is used in FC fabrics. When a packet is sent, it costs 1 credit on the sending side. This credit is
returned to the pool when an acknowledgement is received from the other side. When the switch runs
out of buffer credits, it will not send any new packets on the interface until it receives
acknowledgements.
The maximum number of credits is very much dependent on the switch type and the line card that is
used for the interface. There are many configuration options for sharing credits in an ASIC between
different ports.
The recovery that is asked for has to do with the fact that sometimes the acknowledgements might get
lost on longer interfaces due to bit errors. This means that in time, the amount of buffer credits get’s
less on the interface, while this is not right, as the receiving switch has space enough. This is called the
recovery of credits to get them back periodically. This process is disabled by default.
Be aware that the credits that you configure are transmitted to the other end of the interface. The
buffer you configure is the receive buffer, this receive buffer is then communicated so the other side
knows how much buffer space it can fill up by sending frames.
We can verify this by checking the hardware options of the interface and in which port-group it’s
located:
Port-Group 1
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 8.8 Gbps
Allocated dedicated bandwidth is 4.0 Gbps
--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode
Buffers (Gbps)
--------------------------------------------------------------------
fc1/1 16 4.0 shared
fc1/2 16 4.0 shared
fc1/3 16 4.0 shared
fc1/4 16 4.0 shared
fc1/5 250 4.0 dedicated
fc1/6 16 4.0 shared
Port-Group 2
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 12.8 Gbps
Allocated dedicated bandwidth is 0.0 Gbps
MDS1#
Interface fc1/1 is located in port-group one. In this case, the amount of buffer credits is limited to the
module. Some other modules have limitations per port-group.
In this case we have 3765 credits to spare on an interface. The problem is that we can only use up to
250 credits for E-ports. We can however use Extended B2B credit buffers for this use. When
the regular credits run out, we are able to use the extended buffers up to the maximum. Finally we
disable the credit recovery.
You require the Enterprise license to activate the Extended B2B credit buffers. Remind
that you need to enable the feature separately.
MDS1
MDS1(config)# feature fcrxbbcredit extended
MDS1(config)# int fc1/1
MDS1(config-if)# switch fcrx
fcrxbbcredit fcrxbufsize
MDS1(config-if)# switch fcrxbbcredit ?
<1-500> Enter receive BB_credit
default Default receive BB_credit
We can’t configure that much buffers as we just don’t have them available. We configure the maximum
that remains after subtracting all other interface credits.
As the final step we enable credit recovery, by enabling the BB_SC_N field towards the other side.
Now we verify that we are indeed using that much credits on the interface. As we can see the other end
communicated 250 credits to our interface.
version 5.2(6)
interface fc1/1
switchport rate-mode dedicated
switchport fcrxbbcredit extended 4012
switchport mode E
no shutdown
Port-Group 1
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 4.8 Gbps
Allocated dedicated bandwidth is 8.0 Gbps
--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode
Buffers (Gbps)
--------------------------------------------------------------------
fc1/1 4012 4.0 dedicated
fc1/2 16 4.0 shared
fc1/3 16 4.0 shared
Port-Group 2
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 12.8 Gbps
Allocated dedicated bandwidth is 0.0 Gbps
MDS1(config-if)#
The Maximum Transmission size is also a configuration item that is communicated to the other
end. Meaning that you do not configure the Maximum Transmission Size on the interface, but
you configure the Maximum Receive Size of the packet, because you (as in the switch) know how
much buffer space you have to receive Fibre Channel frames. The ‘MTU’ size is by default configured to
the maximum supported in FC environments which is 2112 bytes.
MDS1
MDS1(config-if)# int fc1/5
MDS1(config-if)# switch fcrxbufsize ?
<256-2112> Enter receive buffer size
As a final task we need to configure State Change Numbers on all JBOD interfaces. This field is
carried in the same BB_SC_N field as the credit recovery is handled. So we need to enable this on all
the JBOD connected interfaces.
MDS1
MDS1(config-if)# int fc1/5
MDS1(config-if)# switch fcbbscn
MDS1(config-if)# int fc1/6
MDS1(config-if)# switch fcbbscn
MDS2
MDS2(config-if)# int fc1/5
MDS2(config-if)# switch fcbbscn
MDS2(config-if)# int fc1/6
MDS2(config-if)# switch fcbbscn
Task 2: VSANs
The next task is regarding the virtualization feature that is supported on Cisco’s MDS switches. This
feature enables the use of different Fibre Channel SANs on the same physical hardware.
This is done using a tagging mechanism which is very similar to 802.1Q tagging in the Ethernet world.
The technical similarity to a VLAN + a VRF in the Ethernet world. As there is a full Layer 2 separation
between the different Fibre Channel Fabrics. In Ethernet terms we would also say that there is a Layer 3
separation as the routing protocol (FSPF) is also separated.
During this task we will be using the VSANs and the routing protocol FSPF to steer traffic across
different ways through our fabric. Another topic in this task is port-channeling of ISL links.
All settings on the MDS are VSAN dependent. Therefore there is a total separation between control-
planes of the VSANs. This introduces a way that you need to specify the VSAN in each of the show and
configuration commands that you type in.
This also means that Zoning in one VSAN has absolutely no impact on Zoning in other VSANs.
The drawing below illustrates the use of VSANs, by using the trunking between MDS switches, you can
connect devices in any VSAN on any MDS switch and transport the FC frames between switches by using
the EISL tagging (Enhanced ISL) on interfaces.
Throughout this task explanation, topics will be explained during the tasks that are discussed.
The first tasks are regarding the creation of VSANs and we should place some interfaces into the VSANS.
Just like VLANs, interfaces are placed inside a VSAN.
MDS1
MDS1(config)# vsan database
MDS1(config-vsan)# vsan 10 name IPX_VSAN_10
MDS1(config-vsan)# vsan 10 interface fc1/5
MDS1(config-vsan)# vsan 20 name IPX_VSAN_20
MDS1(config-vsan)# vsan 20 interface fc1/6
MDS1(config-vsan)# vsan 30 name IPX_VSAN_30
MDS1(config-vsan)# vsan 40 name IPX_VSAN_40
MDS2
MDS2(config)# vsan database
MDS2(config-vsan)# vsan 10 name IPX_VSAN_10
MDS2(config-vsan)# vsan 10 interface fc1/6
MDS2(config-vsan)# vsan 20 name IPX_VSAN_20
MDS2(config-vsan)# vsan 20 interface fc1/5
MDS2(config-vsan)# vsan 30 name IPX_VSAN_30
MDS2(config-vsan)# vsan 40 name IPX_VSAN_40
By default interfaces are placed into a VSAN. When you want to support the moving of certain hosts
throughout the fabric, you can use a technology called Dynamic Port VSAN Membership
(DPVM). This technology will place a port into a VSAN when a certain PWWN is seen.
The DPVM technology can also be distributed throughout the fabric, which is enabled by default, so that
it’s independent of the switch where it comes online and it’s automatically placed into the correct VSAN.
You can also enable the auto-learn feature which enables the automatic learning of all devices coming
online on interfaces. These are then placed into the DPVM database and the next time they come online
on another interface, the interface is placed into that new VSAN.
The DPVM configuration overrules the interface VSAN configuration, so the DPVM database is leading in
this case.
Keep in mind that the only port where this works on is F-ports.
We start with the 2 tasks that place either a WWPN or a Device-Alias in the database for DPVM.
MDS2
MDS2(config)# feature dpvm
Enable DPVM first on the second switch. If you commit the changes on one switch and enable it later on
another switch, you need to commit again. So pay attention to when you configure things that are
distributed. This feature and the distribution of it, needs to be enabled on all switches first, before
information is actually synchronized.
Copyright © by IPexpert. All rights reserved. 228
CCIE Data Center Lab Preparation Detailed Solution Guide
MDS1
MDS1(config)# feature dpvm
MDS1(config)# dpvm distribute
MDS1(config)# dpvm database
MDS1(config-dpvm-db)# pwwn 20:11:00:0a:31:00:aa:de vsan 30
MDS1(config-dpvm-db)# device-alias J1D1 vsan 40
MDS1(config-dpvm-db)# dpvm activate
MDS1(config)# dpvm commit
Now we can see that all information is synchronized between the switches. By using the ‘status’
command you can always check the configuration of the CFS services running for this particular feature.
Very useful when troubleshooting problems with the CFS distribution.
MDS2
MDS2(config)# do sh dpvm status
No active DB, auto-learn is off, distribution is enabled,
Duplicated pwwn will be Rejected.
MDS2(config)#
After configuring MDS1, we see that everything is synchronized between the switches
The next questions are related to the load balancing that is used within a VSAN to distribute information
across the fabric. This load balancing mechanism is used for both Equal Cost Multi Pathing
(ECMP) when 2 equal costs exist in the FSPF database or when a Port-channel is used with
multiple links bundled as one logical interface.
This method ensures that flows between a source and destination pair is always using the same
link
The OX-ID is a value in the Fibre Channel header that is used to identify a traffic flow. Every
‘exchange’, or every ‘action’, done between 2 devices gets a new value attached. All these
values are separated across different hashes so this mechanism achieves a much more granular
load balancing.
In this task VSAN 10 should use the basic Source and Destination FCID where as VSAN 20
should use exchange based load balancing. This is configured using 2 relatively easy
commands on both switches:
MDS1
MDS1(config)# vsan database
MDS1(config-vsan)# vsan 10 loadbalancing src-dst-id
MDS1(config-vsan)# vsan 20 loadbalancing src-dst-ox-id
MDS2
MDS2(config)# vsan database
MDS2(config-vsan)# vsan 10 loadbalancing src-dst-id
MDS2(config-vsan)# vsan 20 loadbalancing src-dst-ox-id
All ISLs should then be made possible to transport all the different VSANs that we created. This means
we now need to enable the trunking on the ISLs to enable this.
MDS1
MDS1(config-if)# interface fc1/1
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# interface fc1/2
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# interface fc1/3
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# interface fc1/4
MDS1(config-if)# switchport trunk mode on
MDS2
MDS2(config-if)# interface fc1/1
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# interface fc1/2
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# interface fc1/3
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# interface fc1/4
MDS2(config-if)# switchport trunk mode on
To verify the working of the trunks, we can issue a similar command to what we use in the Ethernet
world.
Vsan 20 is up (None)
Vsan 30 is up (None)
Vsan 40 is up (None)
fc1/3 is trunking
Vsan 1 is up (None)
Vsan 10 is up (None)
Vsan 20 is up (None)
Vsan 30 is up (None)
Vsan 40 is up (None)
fc1/4 is trunking
Vsan 1 is up (None)
Vsan 10 is up (None)
Vsan 20 is up (None)
Vsan 30 is up (None)
Vsan 40 is up (None)
MDS1(config)#
The next questions are regarding the bundling of interfaces in single logical bundles. This is where the
load balancing is used between the different members of port-channels.
Port-channeling on the MDS switches is also a unique feature of the Cisco MDS platform. These
port-channels have equal options as the standard FC interfaces.
The port-channels also support a protocol to negotiate there possibilities, just like LACP in the 802.3ad
protocol for Ether Channels. Almost the same rules are used for including interfaces in a bundle. The
speed, ASIC rate-mode, BB credits, Trunking or Non-Trunking mode and allowed
VSANs need to be equal.
It’s possible to channel both E (TE) and F (TF) ports, when the platforms support it. One place where
this is commonly seen is in environments that uses N-Port Virtualization (NPV).
We start by configuring the first port-channel between the 2 MDS switches. This is done in a relatively
equal way to Ethernet port-channels on the Nexus switches.
MDS1
MDS1(config-if)# interface port-channel 101
MDS1(config-if)# switchport mode e
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/1
MDS1(config-if)# channel-group 101
fc1/1 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS1(config-if)# int fc1/1
MDS1(config-if)# no shut
MDS1(config-if)# interface fc1/3
MDS1(config-if)# channel-group 101
fc1/3 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
MDS2
MDS2(config-if)# interface port-channel 101
MDS2(config-if)# switchport mode e
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/1
MDS2(config-if)# channel-group 101
fc1/1 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS2(config-if)# int fc1/1
MDS2(config-if)# no shut
MDS2(config-if)# interface fc1/3
MDS2(config-if)# channel-group 101
fc1/3 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS2(config-if)# int fc1/3
MDS2(config-if)# no shut
Now we brought up our first port-channel. Keep in mind that you might need to shut and no-shut
certain interfaces as the ports might get suspended when you are configuring the other switch for the
port-channeling.
MDS1(config-if)#
Now we also need to make them auto negotiate their bundling capabilities. These capabilities activate a
protocol similar to LACP. Instead of configuring it under the member interfaces, this mode is changed
under the port-channel configuration.
MDS2
MDS2(config-if)# int port-channel 101
MDS2(config-if)# channel ?
mode Set the channel mode for the port-channel interface
MDS2
MDS1(config-if)# do sh port-channel database
port-channel 101
Administrative channel mode is on
Operational channel mode is on
Last membership update succeeded
First operational port is fc1/1
2 ports in total, 2 ports up
Ports: fc1/1 [up]
fc1/3 [up] *
MDS1(config-if)#
The next task is asking about another channel. In the past the MDS switches knew a mechanism which
allowed the auto creation of port-channels. You could enable a single command on a range of interfaces
and the switch then automatically detects which ports are eligible for bundling towards the same other
MDS switch on the other end.
This creates a very easy way to configure link bundles. You could also configure the interface after it’s
been created with changes to the whole bundle. The number that was used for the channel depends on
the maximum number of supported port-channels on that particular platform, as it uses the last
available number.
However, as of NX-OS 4.1(1b), this feature was removed as it also introduced issues in that you didn’t
know where the port-channels were pointing to and you really had to know what was going on the
switches.
MDS1
MDS1(config-if)# interface port-channel 127
MDS1(config-if)# switchport mode e
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/2
MDS1(config-if)# channel-group 127
fc1/2 added to port-channel 127 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS1(config-if)# int fc1/2
MDS1(config-if)# no shut
MDS1(config-if)# interface fc1/4
MDS1(config-if)# channel-group 127
fc1/4 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS1(config-if)# int fc1/4
MDS1(config-if)# no shut
MDS2
MDS2(config-if)# interface port-channel 127
MDS2(config-if)# switchport mode e
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/2
MDS2(config-if)# channel-group 101
fc1/2 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS2(config-if)# int fc1/2
MDS2(config-if)# no shut
MDS2(config-if)# interface fc1/4
MDS2(config-if)# channel-group 127
fc1/4 added to port-channel 127 and disabled
please do the same operation on the switch at the other end of the port-
channel,
then do "no shutdown" at both ends to bring it up
MDS2(config-if)# int fc1/4
MDS2(config-if)# no shut
Now what remains is that we need to change our trunks to be using different links. The first 2 questions
are related to the allowed VSAN command, which enables which VSANs are allowed to transfer across
the links.
MDS1
MDS1(config-if)# interface port-channel 101
MDS1(config-if)# sw trunk allowed vsan 10
MDS1(config-if)# sw trunk allowed vsan add 20
MDS1(config-if)# sw trunk allowed vsan add 40
MDS1(config-if)# interface port-channel 127
MDS1(config-if)# sw trunk allowed vsan 10
MDS1(config-if)# sw trunk allowed vsan add 20
MDS1(config-if)# sw trunk allowed vsan add 30
MDS2
MDS2(config-if)# interface port-channel 101
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# sw trunk allowed vsan add 40
MDS2(config-if)# interface port-channel 127
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# sw trunk allowed vsan add 30
Now to verify we only see the VSANs that we configured under the interfaces to be trunked to the other
MDS.
The other 2 questions are related to the steering of traffic across the fibre channel fabric. The focus in
this question is about the FSPF protocol that is working on all FC switches. This protocol is very closely
connected to the OSPF protocol and works basically the same. The easy part is that it works fully
automatically. There is no configuration necessary and will immediately start creating neighbors and
exchange information. The only information that is exchanged in the FSPF protocol is the Domain ID
of the switch in the specific VSAN. The Fibre Channel routing uses those FC Domain IDs to steer
traffic to the correct switch. After it reaches the destination switch it will perform a look-up to find the
egress interface of the traffic where it can send its traffic to. This is all bound to a VSAN basis. Therefore
a VSAN can also be called a Layer 3 VRF in the IP world, as the routing protocol is also fully separated.
We can steer the traffic by changing the FSPF link cost between the switches. FSPF link cost is dependent
on the bandwidth on the link. Therefore also for port-channels the amount of bandwidth of all member
interfaces is counted up to a high bandwidth number.
• 1Gbps: 1000
• 2Gbps: 500
• 4Gbps: 250
• 8Gbps: 125
As we are working with 8Gbps links we have a default cost of 125. Therefore we will lower the cost on
the links we want to make active for a certain VSAN.
We need to make VSAN10 to use port-channel101 and VSAN20 to use port-channel 102 as their primary
links.
MDS1
MDS1(config-if)# interface port-channel 101
MDS1(config-if)# fspf cost 99 vsan 10
MDS1(config-if)# interface port-channel 127
MDS1(config-if)# fspf cost 99 vsan 20
MDS2
MDS2(config-if)# interface port-channel 101
MDS2(config-if)# fspf cost 99 vsan 10
MDS2(config-if)# interface port-channel 127
MDS2(config-if)# fspf cost 99 vsan 20
To verify we need to use a little strange command to find out which interface is actually used for the
traffic. By default the command doesn’t show which interface is used. We do see the correct FSPF cost
already being in place.
When we look a little deeper, by specifying the destination FC Domain ID, we can see the actual
interface that is used for the next-hop. There we see that we are having a properly configured FSPF
network!
The next question is regarding the possibility of the FC Fabric to deliver packets in-order between hosts.
Some hosts that are connected to the fabric require this, for example HP/UX hosts or FICON hosts
assume that the fabric will deliver packets in-order and will have various issues when traffic is not
delivered in the same order as it was sent.
Therefore you can enable the in-order delivery of packets per VSAN.
MDS1
MDS1(config)# in-order-guarantee vsan 30
MDS2
MDS2(config)# in-order-guarantee vsan 30
It’s even possible to create static routes in the FC fabric network by using the ‘fcroute’ command. We
will need this kind of traffic engineering for the next question to send traffic between two
specific disks across another link than the default traffic is going.
In this question you should be aware that it’s talking about 2 ways of traffic, so the steering needs to be
set back to disk as well (bidirectional traffic flow).
Pay close attention to the FC Domain ID of the switch and the port. These can change when the host is
re-connected or the host is rebooted and the Domain ID is not statically configured. In this question it’s
not that relevant, but in the lab, keep this in mind. You will want to have static FCID’s configured when
you need to create a configuration like this.
MDS2
MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain
136 ?
aggregate Aggregable route
metric Cost of route
remote Destination switch remotely connected
vsan VSAN id
When you specify a destination Domain ID of anything behind the next-hop switch, you should
specify the remote keyword here. As otherwise the switch will always assume that the domain ID
specified is also the next-hop switch. Look up this domain ID using the ‘show fcdomain vsan’
command.
MDS1(config-vsan-db)#
MDS2
MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain
136 vsan 10
MDS1
MDS1(config)# fcroute 0x7f0000 0xffffff interface port-channel127 domain
127 vsan 10
The final verification is again by using the fcroute show command, so we can check specific
destinations, not only for the Domain ID mask, but also for the host mask.
In this show output we see that all traffic is sent using the FSPF route across port-channel 101
and that traffic specific for the disk in the JBOD is sent across the other port-channel. Just what the
question stated!
In the Fibre Channel network there is also a root switch for all the Multicast and Broadcast traffic that
is sent throughout the fabric. By default the root switch is also the Principal switch of the fabric. In
case of VSAN 20 we need to change this to be the lowest domain ID switch
MDS1
MDS1(config)# mcast root lowest vsan 20
MDS2
MDS2(config)# mcast root lowest vsan 20
To verify we see that by default the Principal switch is chosen as the default multicast root. Now for
VSAN 20 we changed this to be the lowest Domain ID switch.
MDS2(config)#
The next task is regarding so-called iSPF calculations. With these incremental changes the SPF
calculations of the FSPF protocol are done only on the changes.
MDS2
MDS2(config)# fspf config vsan 30
MDS2(config-(fspf-config))# ?
min-ls-arrival Configure Min LS Arrival
min-ls-interval Configure Min LS interval
no Negate a command or set its defaults
region Configure the autonomous region for FSPF
spf Configure parameters related to SPF route computation
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
MDS2(config-(fspf-config))# spf ?
hold-time Configure the hold time between route computations
static Force static spf computation
MDS1
MDS1(config)# fspf config vsan 30
MDS1(config-(fspf-config))# spf static
The final question in this task is that we want unused ports not to use the Default VSAN. The
Default VSAN is always VSAN 1. Now unused ports are by default configured into VSAN 1.
This cannot be changes easily, but we can ensure that there is no traffic being sent within VSAN 1. This
is done by making it ‘suspended’, but pay close attention! This makes the process difficult as all
interfaces start from VSAN 1 including all ISLs.
To overcome this limitation is to configure the ports in another VSAN just to get them coming online.
Keep this also in mind in later chapters where you are configuring additional links in the FC fabric,
like FCIP tunnels, etc.
MDS1
MDS1(config)# vsan database
MDS1(config-vsan-db)# vsan 1 suspend
MDS2
MDS2(config)# vsan database
MDS2(config-vsan-db)# vsan 1 suspend
Now initially everything looks fine and the trunks are still up and running.
However when the links are re-enabled, they will not come back online and you will get into a very tricky
and stressful situation when you are far down the lab!
This problem is overcome by giving the ISL links another VSAN in the VSAN database so they are made
online within another VSAN, before the trunking protocol is initialized.
MDS1
MDS1(config)# vsan database
MDS1(config-vsan-db)# vsan 10 interface port-channel101
MDS1(config-vsan-db)# vsan 10 interface port-channel127
MDS1(config-vsan-db)# interface port-channel101
MDS1(config-if)# shut
MDS1(config-if)# no shutdown
MDS1(config-if)# interface port-channel127
MDS1(config-if)# shut
MDS1(config-if)# no shutdown
MDS2
MDS2(config)# vsan database
MDS2(config-vsan-db)# vsan 10 interface port-channel101
MDS2(config-vsan-db)# vsan 10 interface port-channel127
MDS2(config-vsan-db)# interface port-channel101
MDS2(config-if)# shut
MDS2(config-if)# no shutdown
MDS2(config-if)# interface port-channel127
MDS2(config-if)# shut
MDS2(config-if)# no shutdown
After the re-configuration of the ISL links, the port-channel interfaces are coming online just as they
were with the correct trunking protocol and we are up and running again!
Again this error is not spotted when you are still working on the switches and they are not flapped. Once
they are flapped or the switch is rebooted the links will remain down unless they are given another
interface in the VSAN database like we just changed.
The final question of this task is for using another protocol in our Fibre Channel Fabric.
This protocol is called the IP over Fibre Channel protocol (IPFC), which is very different
from the FCIP protocol!
This protocol is to create IP interfaces across the FC infrastructure. You can use different protocols for
this use, one of them is management of the FC fabric network or for using CFS over IP services.
The configuration is very similar to creating Switched Virtual Interfaces (SVI’s) in the
Ethernet world. Don’t forget to name the VSAN as this is a requirement from the beginning of this task.
MDS1
MDS1(config)# vsan data
MDS1(config-vsan-db)# vsan 50 name IPX_VSAN_50
MDS1(config-vsan-db)# interface port-channel101
MDS1(config-if)# sw trunk allowed vsan add 50
MDS1(config-if)# int port-channel127
MDS1(config-if)# sw trunk allowed vsan add 50
MDS1(config-if)# interface vsan 50
MDS1(config-if)# ip add 198.18.50.1 255.255.255.0
MDS1(config-if)# no shut
MDS2
MDS2(config)# vsan data
MDS2(config-vsan-db)# vsan 50 name IPX_VSAN_50
MDS2(config-vsan-db)# interface port-channel101
MDS2(config-if)# sw trunk allowed vsan add 50
MDS2(config-if)# int port-channel127
MDS2(config-if)# sw trunk allowed vsan add 50
MDS2(config-if)# interface vsan 50
MDS2(config-if)# ip add 198.18.50.2 255.255.255.0
MDS2(config-if)# no shut
After the configuration, we can see that the new VSAN is allowed on the Trunk interfaces and we can
ping the other MDS switch! This means that the IPFC connection is working.
Now task 2 is finished and we continue with the next task about Zoning!
Task 3: Zoning
This third task will be about configuring a typical Fibre Channel feature. By default nobody in the FC
network can communicate to each other! This is a very important topic about Fibre Channel. You can
compare Zoning to Access Lists in the IP world. By default there is a deny rule on all the
interfaces.
The zoning needs to be configured per VSAN. All devices that are configured under a zone are able to
communicate to each other. Devices between zones can’t talk to each other. Zones are combined
under a Zoneset. This zoneset is activated on a specific VSAN and spread throughout the FC Fabric.
When devices are not specified within a zone, they are part of the Default Zone. This Default
Zone will not process any traffic unless specifically configured in the CLI. Devices can be configured
under multiple Zones at the same time. For example a certain server needs access to various JBODs,
but the JBODs can not communicate to each other. This would result in a number of zones each
specifying the same server and 1 JBOD.
The standard zoning was not sufficient for larger environments. This is why Enhanced Zoning was
created. With Enhanced Zoning it becomes possible to perform zoning configuration from a
single switch at the same time. With basic zoning, administrators can override each others changes
easily by just applying a different zoneset to the VSAN. With Enhanced zoning, all information is
distributed within the fabric and the fabric is locked for changes when one administrator starts
configuring zoning.
We will start by configuring relatively easy zoning and will build up to more complex zoning including
advanced features.
One important thing to remember with zoning is that you should never forget to change the zoning
when this is required. When we will be configuring the UCS system later on, we might require some
additional zoning to be done, while this might not be specifically mentioned in the lab task, however
you can not access your storage without any configuration about the zoning.
The first question in this task is by creating a task by using the device-alias of the devices. Since we
already enabled Enhanced Device-Aliases, we will keep seeing these names in the zoning
configurations.
Pick easy to remember names for your zones. You could create zone names with the devices in it, but
usually what’s best to just use a simple number in the names to make it clear. You can have overlapping
zone names between VSANs, as they are never created as VSAN independent.
MDS1
MDS1(config)# zone name ZONE1 vsan 10
MDS1(config-zone)# member device-alias J1D2
MDS1(config-zone)# member device-alias J1D3
MDS1(config-zone)# member device-alias J1D4
MDS1(config-zone)# zoneset name ZONESET1 vsan 10
MDS1(config-zoneset)# member ZONE1
MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 10
Zoneset activation initiated. check zone status
MDS1(config-zone)#
MDS2
MDS2(config)# do sh zoneset active vsan 10
zoneset name ZONESET1 vsan 10
zone name ZONE1 vsan 10
device-alias J1D3
device-alias J1D4
device-alias J1D2
MDS2(config)#
On MDS2 the only thing we need to do is verify, as the zones are distributed throughout the fabric. Pay
attention that only the active zones are distributed! MDS2 does not have the configuration for the
zones in its running-configuration now!
The next task requires you to use something different than de device-aliases. It’s possible to
create zones in many different ways, using various options to include devices.
MDS1
MDS1(config)# zone name ZONE2 vsan 10
MDS1(config-zone)# member ?
device-alias Add device-alias member to zone
domain-id Add member based on domain-id,port-number
fcalias Add fcalias to zone
fcid Add FCID member to zone
fwwn Add Fabric Port WWN member to zone
interface Add member based on interface
pwwn Add Port WWN member to zone
symbolic-nodename Add Symbolic Node Name member to zone
Remember to always activate the zoneset. Without activation the configuration is not working
and devices can still not communicate to each other!
Now when we verify we see that MDS2 can automatically map the configured PWWNs to the device-
alias that is also configured for this JBOD disk.
The next task is to add an interface of another MDS switch to the zoning configuration. This can be
done by using 2 factors. Either the Domain ID or the SWWN of the switch. The SWWN is a given fact of
the switch that is not possible to change, so this would be the preferred way to configure this type of
zoning.
MDS2
MDS2(config)# sh wwn switch
Switch WWN is 20:00:00:05:9b:73:57:40
We first do a lookup of the SWWN of the switch before we can configure the new zone.
MDS1
MDS1(config)# zone name ZONE3 vsan 10
MDS1(config-zone)# member interface ?
fc Fiber Channel interface
san-port-channel SAN Port Channel interface
Next question is regarding the frames that are sent to the Fibre Channel Broadcast address
(0xFFFFFF). By default broadcast frames arrive to all hosts that are configured in zones. You can
limit this by using ‘broadcast zoning’. This broadcast zoning is enabled by using the
‘broadcast’ keyword in the zones. When this keyword is configured in one of the zones in the VSAN,
the broadcasting is immediately limited to only the zones that have the configuration applied.
This would solve our question! Do not forget to enable broadcast zoning first.
MDS1
MDS1(config-zone)# zone broadcast enable vsan 10
MDS1(config)# zone name ZONE2 vsan 10
MDS1(config-zone)# attribute broadcast
MDS1(config-zone)# zoneset activate name ZONESET1 vsan 10
MDS1(config)#
The next question is required to activate your zoning. Best way to do it, would be to activate the
zoneset every time you perform a change. It’s difficult to verify what has been enabled and what
hasn’t. By just activating it after each change, you will never forget to enable zoning. Again you will
loose points on the lab when this hasn’t been done.
The next tasks will let you add another zone to a zoneset, but not change the active zoneset as
well. We still want all of the zonesets and zones to be visible on all switches, which is by default not
done. The process that performs these changes is known as Full Zoneset Distribution.
We can do this in 2 ways, every time a zone distribution is done, this can be performed. In that case the
configuration should be done in the configuration mode, or you can do it only one time, when the
command is issued in the operational mode. To be sure we will do it via the configuration this
time, as the question did not state if it needs to be done only one time or multiple times.
As we can not change any configuration on MDS1 we first need to be sure to have the configuration in
our running-configuration on MDS2. This requires a copy action from the operational CLI first,
before starting to configure zones. Otherwise all configuration on the other MDS switch is overwritten
with the new configuration.
MDS2
MDS2# zone copy active-zoneset full-zoneset vsan 10
WARNING: This command may overwrite common zones in the full zoneset. Do
you want to continue? (y/n) [n] y
MDS2# sh run zone vsan 10
version 5.0(3)N1(1a)
!Full Zone Database Section for vsan 10
zone name ZONE1 vsan 10
member device-alias J1D3
member device-alias J1D4
member device-alias J1D2
MDS2#
Now all zones are configured in the running-configuration of MDS2. Here we can change the
zoning as required for the question.
MDS2
MDS2(config)# zone name ZONE4 vsan 10
MDS2(config-zone)# member fcid ?
<0x0-0xffffff> Enter FCID
MDS2(config-zoneset)# exit
MDS2(config)# zoneset distribute full vsan 10
MDS2(config)# zoneset activate name ZONESET1 vsan 10
Zoneset activation initiated. check zone status
MDS2(config)#
MDS1
MDS1(config)# zoneset distribute full vsan 10
Now when we applied the zoneset, we can check on MDS1 and see that indeed, the non-active
zoneset is transferred as well as is the new zone, which is not active in the active zoneset! We
do need to activate the full zoneset distribution on MDS1 as well, as this setting is not
distributed with the zones.
version 5.0(3)N1(1a)
!Full Zone Database Section for vsan 10
zone name ZONE1 vsan 10
member device-alias J1D3
member device-alias J1D4
member device-alias J1D2
MDS1#
For zoning the next VSAN, we will require to use a new type of zoning. Like previously mentioned a
new way of zoning has been introduced in a newer version of the Fibre Channel standards. This type
of zoning is commonly referred to as ‘Enhanced Zoning’. The standards that describe this are
FC-GS-4 and FC-SW-3. This big advantage of enhanced zoning is that all changes to the zones
are immediately distributed throughout the fabric. You do still need to perform a manual
activation of the zoning after the configuration. A commit of all changes is not enough to make
them active! We also need to use inline zone creation. This is a way that you can configure
zones right out of the zoneset configuration prompt, making it very easy to use and to configure
multiple zones at once.
We start the configuration by making the first zone, which is also asking for some read-only zoning.
Instead of using the attributes right in the zone, Enhanced Zoning requires to use so-called
attribute-groups that can be applied to zones.
MDS1
MDS1(config)# zone mode enhanced vsan 20
WARNING: This command would distribute the zoning database of this switch
throughout the fabric. Do you want to continue? (y/n) [n] y
Set zoning mode command initiated. Check zone status
MDS1(config)#
After the enhanced zoning has been configured, it’s seen on all other switches. This change is
immediately distributed.
MDS2(config)#
Now the zones can be created inline. We are configuring the first zone.
MDS1
MDS1(config)# zone-attribute-group name RO vsan 20
MDS1(config-attribute-group)# read-only
MDS1(config-attribute-group)# exit
MDS1(config)# zoneset name ZONESET1 vsan 20
Enhanced zone session has been created. Please 'commit' the changes when
done.
MDS1(config-zoneset)# zone ?
name Zone name
version 5.0(3)N1(1a)
zone mode enhanced vsan 20
!Active Zone Database Section for vsan 20
zone name ZONE1 vsan 20
member device-alias J2D1
member device-alias J2D2
member device-alias J2D3
MDS1(config)#
The next zoning is using a feature which is dedicated to Cisco MDS switches due to the fact that it
requires Trunk E ports between switches to transport a specific field.
Within Cisco MDS switches it’s possible to use Quality of Service features within zones. By using
these features it’s possible to give certain hosts or JBODs a higher priority over other traffic. In case of
congestion on the TE-ports between the switches, the traffic marked with a high priority gets served
first, after that the medium priority traffic is served and finally the low priority traffic. The QoS settings
are applied per zone. The marking of the traffic is carried in a special field in the EISL header.
We will configure the next zone with the QoS settings and the FWWNs of the disks instead of their
device-alias or PWWN. You can easily find the FWWN with the show fcns database or show
flogi database on the switch where the JBODs are connected.
Don’t forget to use inline zone creation as this was a requirement for the whole of VSAN 20.
MDS1
MDS1(config)# zone-attribute-group name HIGH vsan 20
Enhanced zone session has been created. Please 'commit' the changes when
done.
MDS1(config-attribute-group)# qos ?
priority Priority to be used for frames matching zone
version 5.0(3)N1(1a)
zone mode enhanced vsan 20
!Active Zone Database Section for vsan 20
zone name ZONE1 vsan 20
member device-alias J2D1
member device-alias J2D2
member device-alias J2D3
The next question asks that when devices in VSAN 20 are not configured for zoning, they should still
be able to read data from each other. The only way we can solve this is by allowing traffic between the
hosts that are in the default zone. Besides that they are then immediately able to write data to
each other as well. To stop this, we can change the attributes of the default zone so we can apply
the read-only zoning option as well.
MDS1
MDS1(config)# zone default-zone permit vsan 20
Enhanced zone session has been created. Please 'commit' the changes when
done.
MDS1(config)# zone default-zone vsan 20
MDS1(config-default-zone)# read-only
MDS1(config)# zone commit vsan 20
Commit operation initiated. Check zone status
The next topic is regarding a special feature called LUN zoning. This means that when you specify
the LUN ID which needs to be reached behind the zoning, only this LUN is reachable. In this task you
are asked to configure this.
Remember for the LUN numbers, that they are in hexadecimal notation!
MDS1
MDS1(config)# zoneset name ZONESET1 vsan 20
MDS1(config-zoneset)# zone name ZONE3
Enhanced zone session has been created. Please 'commit' the changes when
done.
MDS1(config-zoneset-zone)# member device-alias J2D5 lun 0x13
MDS1(config-zoneset-zone)# member device-alias J1D6 lun 0x74
MDS1(config-zoneset-zone)# zoneset activate name ZONESET1 vsan 20
MDS1(config)# zone commit vsan 20
Commit operation initiated. Check zone status
MDS1(config)#
Task 4: FC Domain
The next task is regarding a technology called FC Domain. The FC Domain process is used within the
FC Fabric to ensure unique ID’s are assigned to switches. This Domain ID is then used for
routing the Fibre Channel traffic to other switches. Therefore the Domain ID is a crucial part of the FC
Fabric configuration.
Within a fabric there is a switch chosen to be the Principal switch. This principal switch is
then responsible for handing out the Domain IDs to other switches in the fabric, when they are set to
automatically configuring the Domain ID. The Principal switch is selected based on a
configured priority value and as a tie-breaker the lowest SWWN is used.
By using priority values or lowest SWWN a Principal switch of the entire Fibre Channel Fabric
(per VSAN) is chosen.
Domain ID distribution
The principal switch ensures that there are only unique Domain IDs in the fabric. Switches can
statically configure their own Domain ID, but they always need to ask the Principal switch if
this is still a unique value in the fabric. When there are duplicate Domain ID’s the network will not
work!
When switches are configured to automatically get a Domain ID assigned, the Principal switch
will do this.
FC ID allocation
When Domain IDs are allocated to the switches. The switches can start by assigning FCID blocks
(areas) to the connected devices. They base this on their allocated domain ID. The first byte of the
FCID is the Switch Domain ID. The second byte is the Area code and the last byte is the host
address
Fabric reconfiguration
A domain restart is crucial when new FC Domain IDs are selected for the FC Fabric. This restart is
disruptive for the traffic. All traffic is stopped until the FC Domain and FSPF protocol are fully
converged.
The first tasks are all regarding VSAN 10. MDS1 should be the principal switch and have a static
ID configured. The second switch should prefer a Domain ID.
Pay attention to Decimal and Hexadecimal numbers in these tasks. It depends on the show
command if the Domain ID is presented in decimal or in hexadecimal notation.
By default the FC Domain priority is set to 128. When a principal switch is elected, it changes it’s
‘running’ priority to 2. This is to ensure that it remains the principal switch, even when a
new switch comes into the fabric with a lower SWWN. When we are configuring priority values, we
only need to go under the configured priority of the switches as those are the priorities that
are used in times of convergence of the fabric in the VSAN.
MDS2
MDS2(config)# fcdomain domain ?
<0-239> Configure the domain id in decimal, 0 not allowed for static
dom
<0x0-0xef> Configure the domain id in hexadecimal
We require a Disruptive restart before we will be seeing any results of our configuration
changes.
MDS1
MDS1(config)# sh fcdomain vsan 10
The local switch is the Principal Switch.
MDS2
MDS2(config)# sh fcdomain vsan 10
The local switch is a Subordinated Switch.
We can see that MDS1 is the principal switch of the fabric with the correct priority configured. The
Domain ID’s also match with what the question described. It’s also shown that the Domain ID’s can
be configured in either decimal or hexadecimal format, like they are shown in the show command
as well.
Next we need to ensure that Domain IDs are handed out in a sequential order. By default
Domain ID’s are handed out in any order. This change is only necessary on the Principal switch as this
switch is handing out the Domain ID’s.
MDS1
MDS1(config)# fcdomain contiguous-allocation vsan 10
MDS1(config)# fcdomain disruptive restart vsan 10
The disruptive restart is a very tough restart. All traffic in the VSAN in the Fabric is interrupted
when this occurs. Therefore there has been made a security mechanism to prevent this from happening
from each switch in the network. This feature enables that only a few switches in the network can
trigger the disruptive restart.
When you configure different Domain IDs in your network a disruptive restart is necessary
to make them active in the fabric. This is because the Domain IDs are used for allocating FCIDs to
the connected hosts. When the Domain ID changes, these have to change as well.
Therefore we limit the disruptive restarts on MDS1. This means that MDS2 can never issue a
disruptive restart for the whole FC Fabric in VSAN 10.
MDS1
MDS1(config)# fcdomain rcf-reject vsan 10
MDS1(config)# fcdomain disruptive restart vsan 10
The next task is regarding a fixed allocation of a certain device connected to MDS1. JBOD1 is connected
to MDS1 in VSAN 10, where this can get assigned static FCIDs.
The ID’s that need to be assigned are in the range of the Domain ID that is running on the switch. The
only way to statically configure the FCID’s is when they match with the running Domain ID of
course.
MDS1
MDS1(config)# fcdomain fcid database
MDS1(config-fcid-db)# vsan 10 wwn c6:83:00:11:0d:18:fa:ce fcid 0x222200
area
Now by specifying the WWN of J1D1 and the FCID that we want we can specify the area keyword
behind it. This means that it will assign this entire area to that particular WWN and allocate all FCID’s
just for this disk.
When this disk will perform a new FLOGI to the switch, it will get this FCID assigned!
The next question will assign Domain IDs to the switches in VSAN 20. Again pay attention to the
hexadecimal notation of the allowed FC Domain IDs that can be handed out by the Principal
Switch. In this case MDS2 should be the Principal switch. MDS1 should be requesting a
domain ID.
MDS1
MDS1(config)# fcdomain domain 214 preferred vsan 20
MDS2
MDS2(config)# fcdomain priority 99 vsan 20
MDS2(config)# fcdomain allowed 176-206 vsan 20
Error: attempt to manually set an allowed domain list that would not
include all the currently assigned and locally configured domains or that
would conflict with existing allowed domain lists.
MDS2(config)# fcdomain domain 176 static vsan 20
MDS2(config)# fcdomain restart disruptive vsan 20
MDS2(config)# fcdomain allowed 176-206 vsan 20
Error: the VSAN is not stable. Please try again after it stabilizes.
MDS2(config)# fcdomain allowed 176-206 vsan 20
Error: attempt to manually set an allowed domain list that would not
include all the currently assigned and locally configured domains or that
would conflict with existing allowed domain lists.
What happens here is that the allowed list can’t be set in the VSAN unless all Domain IDs are
currently compliant with the configuration. This means that we need to change the Domain ID of
MDS1 temporarily because this introduces the issue.
MDS1
MDS1(config)# fcdomain domain 205 static vsan 20
MDS1(config)# fcdomain restart disruptive vsan 20
MDS1(config)# fcdomain domain 214 preferred vsan 20
Once we change the Domain ID of MDS1 back to something that falls within the range that we want
to configure. After the allowed list is configured a final restart of the VSAN is done.
MDS2
MDS2(config)# fcdomain allowed 176-206 vsan 20
MDS2(config)# fcdomain restart disruptive vsan 20
MDS2 looks fine. We see that it is elected as the Principal Switch of the VSAN and that it is
running a Domain ID that falls within the range.
Now how is MDS 1 working, as we have configured a preferred Domain ID which is outside of the
range that is allowed.
The switch is running another Domain ID as the preferred one was not allowed. Therefore the
Principal switch allocated another ID to this switch, which it is using now.
The final question is related to a feature called fast-restart, which has been recently available on
the MDS switches. We need to turn this feature on in VSAN 30. This is one of those features that is
easily looked up in the Documentation on Cisco.com.
MDS1
MDS1(config)# fcdomain optimize fast-restart vsan 30
MDS1(config)#
MDS2
MDS2(config)# fcdomain optimize fast-restart vsan 30
MDS2(config)# fcdomain restart disruptive vsan 30
MDS2(config)#
Port Security
Port-security ensures that there are no rogue hosts getting connected to the Fibre
Channel Fabric. The feature checks if the device connected has an equal PWWN, FWWN or SWWN
otherwise it is rejected.
Port Security can also protect that devices are not connected on different ports. So
devices are only allowed to come online on certain interfaces. This can be one or more.
Fabric Binding
Fabric Binding is a technology used to protect the FC switches in a Fabric. You will need to
have the SWWNs configured of all switches before it may come online in the fabric. You can
choose to even protect the Domain IDs configured on these SWWNs
FC-SP / DH-CHAP
FC-SP is a protocol which is also called DH-CHAP. Configuration commands all start with
DHCHAP. This feature ensures that all connected devices on the enabled interfaces are
authenticated on the switch before they are allowed access. They need to use a pre-defined
password (either local or via RADIUS) that needs to match before the log-in is allowed on
the switch.
The port-security feature is the first feature that is configured. The initial task is using a specific
feature called auto-learning on port-security. This means that all devices that are connected
to the VSAN are automatically added to the port-security database and saved.
The port-security database can be distributed throughout the Fibre Channel fabric as well, so all auto-
learned entries are saved to all switches.
In the initial task you are not aware of the WWNs in the fabric so you will require using the auto-
learning feature.
MDS1
MDS1(config)# port-security enable
MDS1(config)# port-security auto-learn vsan 10
Error for VSAN 10: No Active database
MDS1(config)# port-security activate vsan 10
MDS1(config)# port-security auto-learn vsan 10
Now we enabled the port-security in VSAN 10 with auto-learning enabled. The next step is to
ensure that it is seen in the configuration of the device. This requires a command on the operational
prompt to copy the auto-learnt entries into the configuration.
MDS1
MDS1(config-if)# do sh port-security data v 10
--------------------------------------------------------------------------
------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------
------
MDS1(config-if)# end
MDS1# port-security database copy vsan 10
MDS1# sh port-sec data
Copyright © by IPexpert. All rights reserved. 261
CCIE Data Center Lab Preparation Detailed Solution Guide
--------------------------------------------------------------------------
------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------
------
10 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
[Total 6 entries]
MDS1#
This copy action is a manual action. It makes it very easy for the initial roll-out of port-security to
have it enabled first and then disabled after the initial learning. Then you can add and remove things in
the database manually.
Don’t forget to perform this same action on MDS2, as there is no distribution enabled in this
question.
MDS2
MDS2(config)# port-security enable
MDS2(config)# port-security activate vsan 10
MDS2(config)# port-security auto-learn vsan 10
MDS2(config)# end
MDS2# port-security database copy vsan 10
After it’s enabled we see the auto learned entries showing up in our database.
The next task will prepare you for new devices that will be connected into the fabric. The port-security
feature also knows the concept of wild-cards. You can enable certain characteristics of a host in the
database where you are unsure on which interface it will be connected. This means you can leave it
completely blank or you can add only a switch where it is added.
As we don’t have any distribution of the database entries between the MDS switches, we do have an
overlap in this question as one of the PWWNs can be connected to either MDS switch.
MDS1
MDS1(config)# port-sec database vsan 10
MDS1(config-port-security)# ?
Port Security Lists:
any-wwn Any Node/Port/Switch WWNs
device-alias N_Port Device_Alias
do EXEC command
end Exit from configure mode
exit Exit from this submode
no Negate a command or set its defaults
nwwn N_Port Node-WWN
pwwn N_Port Port-WWN
swwn Switch-WWN
MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 ?
fwwn Swith-Port WWN
interface Switch Interface name
swwn Switch WWN
<cr> Carriage return.
MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 interface fc1/11
MDS1(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32
After the 2 entries have been added we see in the database that the entries area added and that for the
entry which has an interface configured the SWWN of MDS1 is added to it.
10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
10 20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)*
10 20:00:00:a3:fe:00:98:32(pwwn) Any
[Total 8 entries]
MDS1(config)# do sh wwn sw
Switch WWN is 20:00:00:0d:ec:6f:4f:40
MDS1(config)#
MDS2
MDS2(config)# port-security data vsan 10
MDS2(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32
MDS2(config-port-security)# do sh wwn sw
Switch WWN is 20:00:54:7f:ee:05:08:78
MDS2(config-port-security)# pwwn 20:00:00:A3:DE:11:66:2B swwn
20:00:54:7f:ee:05:08:78
Verification about the just configured parameters. It’s shown that one of the PWWNs can only be
connected to the SWWN of MDS2 and the other entry can be connected anywhere in the fabric.
The next question in this task is to secure VSAN 20 and to manually configure all port-security
parameters. This time all parameters need to be distributed between the switches, because we can only
change MDS1 to finish this task. When we need to be as specific as possible, it means that we need to
connect PWWNs to their respective interfaces. This is the most specific we can get in a port-security
environment.
We first need the FLOGI information of both of the switches so we can create a database using this
information.
MDS1
MDS1(config-port-security)# do sh flogi database vsan 20
--------------------------------------------------------------------------
-
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------
-
fc1/3 20 0x0a00e8 c6:80:00:11:0d:07:fa:ce
c5:9a:00:06:2b:01:00:07
fc1/3 20 0x0a00ef c6:80:00:11:0d:01:01:07
c5:9a:00:06:2b:01:00:07
fc1/4 20 0x0a01e8 c6:81:00:11:0d:17:fa:ce
c5:9a:00:06:2b:02:00:07
fc1/4 20 0x0a01ef c6:81:00:11:0d:02:01:07
c5:9a:00:06:2b:02:00:07
MDS2
MDS2(config)# sh flogi data v 20
--------------------------------------------------------------------------
------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------
------
fc1/1 20 0x0c0000 21:01:00:e0:8b:3c:c0:b5
20:01:00:e0:8b:3c:c0:b5
MDS1
MDS1(config-port-security)# port-security database vsan 20
MDS1(config-port-security)# pwwn c6:80:00:11:0d:07:fa:ce interface fc1/3
MDS1(config-port-security)# pwwn c6:80:00:11:0d:01:01:07 interface fc1/3
MDS1(config-port-security)# pwwn c6:81:00:11:0d:17:fa:ce interface fc1/4
MDS1(config-port-security)# pwwn c6:81:00:11:0d:02:01:07 interface fc1/4
MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 ?
fwwn Swith-Port WWN
interface Switch Interface name
swwn Switch WWN
<cr> Carriage return.
MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 swwn
20:00:54:7f:ee:05:08:78 interface fc1/1
MDS1(config)# port-security activate vsan 20
MDS1(config)# port-security commit v 20
After activating the database on the MDS1, we can see an activated database on both MDS switches
with all necessary information for activating the security in VSAN 20.
MDS1
MDS1(config)# do sh port-security data v 20
--------------------------------------------------------------------------
------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------
------
20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*
[Total 5 entries]
MDS1(config)#
MDS2
MDS2(config)# do sh port-security data v 20
--------------------------------------------------------------------------
------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------
------
20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40*
20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40*
20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40*
20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40*
20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*(fc1/1)
[Total 5 entries]
MDS2(config)#
The last entry on MDS2 shows that it sees the interface that is configured on MDS1 back on MDS2. Our
port-security with distribution works!
Next up is Fabric Binding! This technology is used to protect Fibre Channel fabrics of rogue
switches to log-in to the fabric. It’s also possible to create this kind of protection by using Port
Security. You can specify all of the SWWNs that are allowed access to the Fibre Channel fabric.
However the fabric-binding technology is more useful for this, because it can also use the Domain ID
for securing the fabric.
The initial task is to secure VSAN 30 so only MDS1 and MDS2 are allowed. Because we should also add
the Domain IDs in this task, we should make these static as a reboot might change the Domain ID
of a switch, which might cause the fabric to reject the switches and you might end up with loosing points
for this section.
The configuration must be done from both switches as fabric-binding cannot be distributed
between switches.
MDS1
MDS1(config)# sh wwn sw
Switch WWN is 20:00:00:05:9b:70:5a:00
MDS1(config)# fabric-binding data vsan 30
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1
MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2
MDS1(config-fabric-binding)# exit
MDS1(config)# fabric-binding activate v 30
MDS2
MDS2(config)# sh wwn sw
Switch WWN is 20:00:54:7f:ee:09:3e:c8
MDS2(config)# fabric-binding data vsan 30
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1
MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2
MDS2(config-fabric-binding)# exit
MDS2(config)# fabric-binding activate v 30
MDS1
MDS1(config)# sh fabric-binding data
--------------------------------------------------
Vsan Logging-in Switch WWN Domain-id
--------------------------------------------------
30 20:00:00:05:9b:70:5a:00 0x1(1) [Local]
30 20:00:54:7f:ee:09:3e:c8 0x2(2)
[Total 2 entries]
MDS2
MDS2(config)# sh fabric-binding data
--------------------------------------------------
Vsan Logging-in Switch WWN Domain-id
--------------------------------------------------
30 20:00:54:7f:ee:09:3e:c8 0x2(2) [Local]
30 20:00:00:05:9b:70:5a:00 0x1(1)
[Total 2 entries]
MDS2(config)# sh int trunk vsan
port-channel101 is trunking
Vsan 1 is up (None)
Vsan 10 is up (None)
Vsan 20 is up (None)
Vsan 30 is up (None)
Vsan 40 is up (None)
port-channel127 is trunking
Vsan 1 is up (None)
Vsan 10 is up (None)
Vsan 20 is up (None)
Vsan 30 is up (None)
Vsan 40 is up (None)
MDS2(config)#
When the VSAN is still up on the trunk interfaces we know that the fabric-binding is properly
configured!
The final topic in this security task is the FC-SP feature. This feature enables the authentication of host-
switch and switch-switch interfaces. It combines a CHAP authentication with a Diffie-Hellman
exchange. The steps for configuring FC-SP can become quite complex as a lot of different combinations
are possible.
An important topic is the combination of interface modes. You can configure an interface with different
modes as it might be required that you configure an interface with optional authentication. The table
below demonstrates the different modes and what will happen when these 2 interfaces are connected
to each other.
The On and Off modes are simple modes that do what they tell. They enforce authentication or reject
it. The auto modes are there for optional authentication. There should be at least 1 active link when
both sides are on auto as the passive link will wait for an authentication request before starting any
negotiation.
Behind the mode you have the option to configure the re-authentication timeout, so the authentication
is performed a couple of times and is not done only once in the lifetime of the interface.
MDS1
MDS1(config)# feature fcsp
MDS1(config)# fcsp dhchap dhgroup 4
MDS2
MDS2(config)# feature fcsp
MDS2(config)# fcsp dhchap dhgroup 4
The next step is to configure the password that you want to accept on the switch for authentication
requests. This can be configured as one key per peer device.
This means that when you have several devices connected to the switch you can configure 1 key per
connected device, by specifying the WWN of the switch.
In our example we need to be as specific as possible, although we only have 2 devices, we will still
configure that this key is only accepted for that particular MDS switch that is connected.
MDS1
MDS1(config)# fcsp dhchap ?
devicename Configure password of the remote devices
dhgroup Configure DHCHAP Diffie-Hellman Group List (In preference)
hash Configure DHCHAP Hash Algorithm List (In order of
preference)
password Configure local DHCHAP password
MDS2
MDS2(config)# fcsp dhchap password IPexpertMDS2 20:00:00:05:9b:70:5a:00
The password command is used for the accepted password on the local switch, whereas the
devicename command will configure the password that is used to authenticate the neighbor, so
the sending password.
MDS1
MDS1(config)# fcsp dhchap device 20:00:54:7f:ee:09:3e:c8 password
IPexpertMDS2
MDS2
MDS2(config)# fcsp dhchap devicename 20:00:00:05:9b:70:5a:00 password
IPexpertMDS1
Now the passwords are configured and we can start the first authentication. MDS2 should not initiate
any request, so we will make this a passive link. MDS1 should set the link down when the switch
doesn’t respond. As described in the table above, the only option that makes a link go down is the
‘on/off’ modes. So we are going to use this in this first task.
MDS1
MDS1(config)# interface fc1/1
MDS1(config-if)# fcsp on ?
<CR>
<0-100000> Provide reauthentication interval in minutes
MDS1(config-if)# fcsp on 15
MDS2
MDS2(config-if)# int fc1/1
When the FC-SP protocol is configured and the link is re-initialized, the authentication is working!
MDS1
MDS1(config-if)# sh fcsp interface fc1/1
fc1/1:
fcsp authentication mode:SEC_MODE_ON
reauthentication timeout (in minutes):15
Status:Successfully authenticated on Thu Oct 18 11:57:28 2012
Authenticated using local password database
MDS2
MDS2(config-if)# sh fcsp interface fc1/1
fc1/1:
fcsp authentication mode:SEC_MODE_AUTO_PASSIVE
Status:Successfully authenticated on Thu Oct 18 11:56:20 2012
Authenticated using local password database
The next link to authenticate needs to use SHA1 hashes with a fall-back to MD5. By default this is the
other way around. MD5 hashes are used by default with a fall-back to SHA1.
These are changed for the entire switch at the same time. We will configure both switches to respond
actively to authentication requests.
MDS1
MDS1(config-if)# fcsp dhchap hash sha1 md5
MDS1(config)# interface fc1/2
MDS1(config-if)# fcsp auto-active
MDS2
MDS2(config-if)# fcsp dhchap hash sha1 md5
MDS2(config-if)# int fc1/2
MDS2(config-if)# fcsp auto-active
MDS2(config-if)#
On the third link, which is the second member of port-channel 101 (fc1/3) we will completely
disable FC-SP, which is an easy thing to do. Keep in mind that ports have FC-SP auto-passive
enabled by default!
MDS1
MDS1(config)# interface fc1/3
MDS1(config-if)# fcsp off
MDS2
MDS2(config-if)# int fc1/3
MDS2(config-if)# fcsp off
MDS2(config-if)#
And finally we configure authentication on the last link between the switches. One side can be
configured passive and one side as active. They will still come online when neither link authenticates,
but we do want authentication to happen. Therefore we need at least one end to be active.
MDS1
MDS1(config)# interface fc1/4
MDS1(config-if)# fcsp auto-active
MDS2
MDS2(config-if)# int fc1/4
MDS2(config-if)# fcsp auto-passive
MDS2(config-if)#
FC-SP is always configured under a port-channel member interface and never on the port-channel
itself!
Other places where FC-SP can be configured is on FCIP interfaces. FC-SP is never VSAN dependent.
The initial task is about configuring the distribution of features between switches. This distribution is
done using a Cisco proprietary protocol called Cisco Fabric Services (CFS), which has been
talked about earlier. It’s possible with CFS to configure so-called ‘regions’. The regions that are
configured mean that you can create some MDS switches that have different configuration sets than
other MDS switches.
This is done by configuring the features in their own matching region number. By default every
feature resides in region 1. Up to 200 different regions can be configured.
In this case we want MDS2 to have it’s own call-home features. We can easily accomplish this by
configuring the call-home feature in a different region. When we then configure call-home (or
any other feature that uses CFS) on MDS1 and the changes are committed, they are not accepted on
MDS2 as a valid configuration and not inserted into the configuration.
The configuration for this feature is relatively easy, but it could introduce a lot of complexity in the MDS
configuration.
MDS2
MDS2(config)# cfs region 100
MDS2(config-cfs-region)# callhome
WARNING: If an Application is moved/assigned to a new region,
its scope is restricted to that region and it ignores all other regions
for distribution or merge.
Are you sure? (y/n) [n] y
The next feature that we need to configure are 2 things at the same time. The MDS switches can create
a list of SCSI targets that you can use in the entire FC fabric. This list can then be handed to the
Storage administrators to demonstrate that the FC network is working fine.
The next feature where we want to use this SCSI target list is a Job Scheduler. The job
scheduler is a unique feature of the NX-OS where you can schedule all CLI configurations to be done
either at a given date and time or by configuring it on a repetitive basis, like in our case where we need
to configure the scheduler to run every 24 hours.
The configuration needs to get used to, but there are no complex technologies involved in this case.
Now we want to save this to the flash disk on the MDS switch. We can do this by using a Linux like
command.
MDS1#
To verify we can check if the schedule is configured correctly and if the commands that we configured
under the job are the correct commands.
==========================================================================
====
MDS1(config)# do sh scheduler schedule
Schedule Name : SCSI_SCHED
--------------------------------
User Name : admin
Schedule Type : Run every day at 0 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
Job Name Last Execution Status
-----------------------------------------------
SCSI -NA-
==========================================================================
====
MDS1(config)#
The final 2 tasks of this chapter are introducing another unique Cisco MDS feature.
We are asked to create failovers between devices where the end-user can never notice any changes in
the Fibre Channel network.
The feature that supports this is SAN Device Virtualization (SDV). This feature is a hidden
command to enable since NX-OS 5.x, but it could very well still be an option to configure on the CCIE
Data Center test.
Once the feature is enabled, all commands are still available like they used to be.
We can create a virtual Fibre Channel device on our network that in the background has 2 devices
connected to it for virtualization purposes. We can configure automatic failover and fallback for this
physical device.
The virtual device is taken into the zoning. So we should create a zone for this use so the users in VSAN
10 can communicate to it.
MDS1
MDS1(config)# sdv virtual-device name JBOD1 vsan 10
MDS1(config-sdv-virt-dev)# device-alias J1D1 primary
MDS1(config-sdv-virt-dev)# device-alias J2D1
MDS1(config-sdv-virt-dev)# attribute ?
failover Configure the failover attributes
One important note is the attribute command where we configure the actual automatic failover and
fallback. By default the SDV device will keep pointing to the PWWN of the primary device, unless the link
command is used to change this. We can make this process automatic, by configuring the automatic
failover and fallback parameters.
When the SDV configuration is committed, a device-alias is automatically created for use within the
zoning.
You successfully completed chapter 5 regarding Cisco MDS Fibre Channel features. The next chapter will
focus on Fibre Channel extension over IP connections!
Do NOT erase configurations on the MDS switches before moving on to the next Chapter!
Chapter 6: Data
Center Storage
Networking
Extension
Chapter 6: Data Center Storage networking Extension is intended to let you be familiar with the
Storage Networking features on the Cisco MDS switches. This chapter will be about configuring IP
features like iSCSI, iSLB and FCIP including the relevant Security features for Fibre Channel extension
across IP connections. We highly recommend to create your own diagram at the beginning of each lab
so you are able to draw on your own diagram, making it much easier when you step into the real lab.
Multiple topology drawings are available for this chapter.
General Rules
• Try to diagram out the task. Draw your own connections the way you like it
• Take a very close read of the tasks to ensure you don’t miss any points during grading!
• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Solutions
With FCIP tunnels you are able to extend the Fibre Channel network across a normal
Layer 3 IP network. This overcomes the distance limitations that you have when
expanding Fibre Channel between data centers, where you could easily run into issues
with B2B credits. Besides that you need a Dark Fiber or DWDM transparent system to
transport Fibre Channel. FCIP overcomes these limitations and narrows down your
requirements for transporting FC between Data Centers to just an IP connection
iSCSI
The iSCSI feature on the Cisco MDS platform is not widely used, but still a great
feature and most important of all tested on the CCIE Data Center lab exam. This feature
makes it able for you to present a Fibre Channel disk to an non-FC host. Meaning that
the Server can talk over it’s IP connection with the iSCSI protocol to the MDS switch
which will present itself as an iSCSI LUN, while in the background you are actually
talking to a Fibre Channel disk.
The iSLB feature is a extension of the iSCSI feature where redundancy, distribution
and load balancing are integrated. You could say iSLB is actually iSCSI 2.0 on the
MDS platform
Be aware that you need the SAN Extension over IP license on the MDS switches to enable
FCIP. For iSCSI you do not require a license. On the MDS 9222i the SAN Extension
license is included!
For this task we will be required to use a Ethernet switch to transport our traffic between our
MDS switches. Our physical topology shows that we have our MDS IP ports connected to the
Nexus 5500 switches. This means that we first need to ensure that we transport several VLANs
between the switches.
We will be using several VLANs during this chapter, which is not strictly necessary, but do let
you practice with the 802.1Q features of the IP interfaces on the line cards. The IP interfaces of
the MDS switch are different than a standard router or switch interface. Keep in mind that
these interfaces are actually just ‘host’ interfaces that let you connect a host device to it. This
means that you can just configure an IP address in the same subnet on multiple interfaces. As
they are just hosts on the network and will never start routing between these ports.
The MDS also support port-channeling of these interfaces in standard Ethernet port-
channels. In this chapter we will not be configuring Ethernet Port-channels on the MDS switch,
but configuration is equal to how that is done on the Nexus switches.
Drawing 1 shows you the physical topology we are using in this chapter.
In the first task we are required to ensure that the MDS switches can communicate over a
number of VLANs. You might still have configuration from a previous task on the Nexus
switches, but this is not conflicting with this lab.
The first task is to configure the VLANs and let the MDS switches have access to it. This means
we need to make sure that both GigabitEthernet interfaces on the MDS switches are configured
as Trunk interfaces on the Nexus 5500s.
Configuration is a piece of cake, as we have already finished extensive chapters regarding the
configuration of NX-OS with Ethernet and IP features.
Configuration in this case will only comprise of the VLANs and Trunk configurations. We do not
need to configure IP routing, where this would be an option for FCIP and iSCSI as well. In this
case it would not add a lot to our complexity and we just want to separate traffic across
different VLANs.
We use Drawing 2 to determine the VLANs that we need.
In this drawing we can see that we should be using VLAN 69 through 72 for the IP features of
the MDS switches. In this chapter we will not be using any ISL links between the MDS switches.
We have pre-configuration from the previous chapter on there, but this is not conflicting with
the tasks that we are about to configure.
We re-configure the interfaces between the MDS switches as we require some other traffic
than FabricPath from the last Chapter where we configured the Nexus switches.
SW2
SW2(config)# vlan 69,70,71,72
SW2(config-vlan)# exit
SW2(config)# interface e1/4
SW2(config-if)# switchport mode trunk
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW2(config)# default interface e1/9
SW2(config)# interface e1/9
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 69-72
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW2(config)# default interface e1/10
SW2(config)# interface e1/10
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 69-72
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW3
SW3(config)# vlan 69,70,71,72
SW3(config-vlan)# exit
SW3(config)# interface e1/4
SW3(config-if)# switchport mode trunk
SW3(config-if)# no shutdown
SW3(config-if)# exit
SW3(config)# default interface e1/9
SW3(config)# interface e1/9
SW3(config-if)# switchport mode trunk
SW3(config-if)# switchport trunk allowed vlan 69-72
SW3(config-if)# no shutdown
SW3(config-if)# exit
SW3(config)# default interface e1/10
SW3(config)# interface e1/10
SW3(config-if)# switchport mode trunk
SW3(config-if)# switchport trunk allowed vlan 69-72
SW3(config-if)# no shutdown
SW3(config-if)# exit
MDS1
MDS1(config)# interface gig1/1
MDS1(config-if)# no shutdown
MDS1(config-if)# exit
MDS1(config)# interface gig1/2
MDS1(config-if)# no shutdown
MDS1(config-if)# exit
MDS2
MDS2(config)# interface gig1/1
MDS2(config-if)# no shutdown
MDS2(config-if)# exit
MDS2(config)# interface gig1/2
MDS2(config-if)# no shutdown
MDS2(config-if)# exit
We do not need to configure any IP addressing right now. The final statement in this task is
regarding the future configurations of IP addressing, where we need to pay attention to the
format that we are going to use. Drawing 2 will illustrate which host IP addresses need to be
used and which VLANs belong to which feature.
So we finished the first task!
Task 2: FCIP
In the second task we will start configuring our FCIP tunnels between the switches. In this
chapter you will be configuring both the FCIP tunnels and their High Availability
features that are available on the Cisco MDS platform.
During your CCIE Data Center lab exam you might see questions about the expansion of your
Fibre Channel network to the other Data Center in the lab topology.
Keep in mind that it’s not possible to transport FCoE across the data centers when you are
using FabricPath and/or OTV. Therefore you will see MDS switches with FCIP features being
used in Cisco best practice designs regarding multiple Data Centers.
The FCIP technology is a standard based solution (FC-BB-2) and compatible with other
vendors FC switches.
The goal of the technology is to extend the Fibre Channel fabric across geographically dispersed
locations.
The only traffic that you are able to transport is E-port type traffic, meaning you can only
transport ISL traffic. The Virtual E-port (VE) that is created has the same features as a
normal E-port. Meaning that you can also enable the Trunking protocol to transport multiple
VSANs.
The FCIP interface is the same interface as a normal FC interface on the MDS switch, meaning
that you have an equal amount of configuration commands available.
The drawing below illustrates the positioning of the FCIP tunnel in the Fibre Channel fabric.
The Host and the JBOD in this drawing do not know that they are interconnected through a
Layer 3 IP network as they just talk plain Fibre Channel throughout the FC fabric. All 4 FC
switches are participating in the same FC Domain and FSPF network. Traffic is routed in the
same way, except that the traffic gets an IP encapsulation instead of an FC encapsulation
between the data centers.
The FCIP tunnel consists of 2 TCP sessions:
Control
The control session is used for all Fibre Channel control packets. This means that all
Class F fibre channel frames are transmitted across this TCP session. This ensures that
control traffic can receive a different QoS treatment throughout the IP network to
ensure a low latency transmission of all control frames.
Data
The second TCP session is used for all other data (Class 1, 2, 3) frames that are
sent across the FC fabric. These data packets can get another QoS marking so they fall in
different classes when required.
You can choose to have only 1 TCP session for the FCIP connection. By default there will be 2
sessions.
Configuration of the FCIP connection consists of 2 sections:
FCIP Profile
The FCIP profile configured the TCP session parameters and the local interface
parameters. This configures the source interface of the FCIP tunnel. A single profile
can be used by multiple tunnels.
FCIP Interface
The FCIP interface configurations is the actual tunnel interface. On this interface all
Fibre Channel related features are configured, a profile is attached, the peer IP
address is configured and the number of TCP sessions used for this tunnel.
To create a FCIP interface you need to have at least 1 differentiation out of these 4
parameters:
• Local IP address
• Peer IP address
• Peer port
You can not create multiple interfaces where all these parameters are equal, as that would
result in the same interface. When one of these 4 is different, the interface can come up, but it
might require some engineering regarding the initiation of the interface.
Our first question of the task is to configure a FCIP connection between the 2 MDS switches.
We will transport only 2 already active VSANs across this interface. The FCIP tunnel is using
only a single interface and using only 1 TCP session between the switches.
We can not use the default TCP port, which is TCP 3225, so we need to change this as well. As
this is a TCP session parameter we need to configure this under the profile configuration.
MDS1
MDS1(config-if)# interface gig1/1.69
MDS1(config-if)# ip address 198.18.69.9 255.255.255.0
MDS1(config-if)# no shutdown
MDS1(config-if)#
MDS1(config-if)# feature fcip
MDS1(config)# fcip profile 1
MDS1(config-profile)# ip address 198.18.69.9
MDS1(config-profile)# port ?
<1-65535> Profile TCP Port
MDS1(config-profile)#
The profile configuration now has the source IP address configured and the non-default port
that we are required to use. Other TCP parameters that can be set are traffic-shaping
parameters and some TCP features. All the advanced TCP features like Path MTU Discovery
(PMTU) and Selective Acknowledgements (SACK) are by default enabled.
The interface that we configure has a sub-interface number of 69. This number is equal to
the VLAN number, because this is immediately the VLAN configuration. There is no
encapsulation command like on normal router/switch interfaces. So keep in mind which
sub-interface number you use for this case as it’s crucial for the VLAN numbering.
The amount of TCP connections is still supported to be configured, except that it’s not available
through the question mark anymore. Therefore you should know that this command is
configured under the FCIP interface with the tcp-connections command. You may
specify a value of 1 or 2 behind it, where 2 is the default.
Now the FCIP interface should be configured.
MDS1
MDS1(config-profile)# interface fcip1
MDS1(config-if)# use-profile 1
MDS1(config-if)# peer-info ipaddr 198.18.69.11 port 3226
MDS1(config-if)# tcp-connections 1
MDS1(config-if)# switchport ?
description Enter description of maximum 80 characters
fcrxbufsize Configure receive data field size for the port
mode Enter the port mode
trunk Configure trunking parameters on an interface
MDS1(config-if)# no shut
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface fcip1
The final configuration that is done in this case, adding the FCIP interface to a VSAN in the
VSAN database, has to do with the fact that one of the tasks in the previous chapter was to
deactivate the Default VSAN (VSAN 1). By making it suspended, every interface that has it’s
registration in VSAN 1 will stay down unless something else is configured. In the case of Trunk
interfaces, the port first needs to come online before the trunking is negotiated with the other
side. When the trunk port is configured in the Default VSAN it will first come online in this
VSAN before it will start trunking. Therefore we need to configure the FCIP interface, as it’s
treated as just another FC interface, inside the VSAN database into another VSAN. The VSAN
that you configure the tunnel in, is not that significant, except that it should be anything else
than VSAN 1.
The FCIP connection is now configured on one side. There is one additional requirement for
this initial few questions of this task, which is that MDS1 should always initiate the connection.
This is achieved using the passive-mode command, where the switch where it’s configured
will never initiate the connection and will always wait on the other side for setting up the
tunnel.
This passive-mode is required when you are configuring multiple FCIP tunnels that each
should terminate on the same FCIP profile, meaning you have only 1 destination IP/port for
the remote side. When this switch with only 1 listening port, would initiate the connections
there is no problem as the switch will initiate the connection correctly to the right switch. When
it receives traffic on the FCIP port, it will not know to which FCIP tunnel the traffic actually
belongs.
In this case it has no special use, except that you know how to configure this.
What remains is to configure the other side.
MDS2
MDS2(config-if)# interface gig1/1.69
MDS2(config-if)# ip address 198.18.69.11 255.255.255.0
MDS2(config-if)# no shutdown
MDS2(config-if)#
MDS2(config-if)# feature fcip
MDS2(config)# fcip profile 1
MDS2(config-profile)# ip address 198.18.69.11
MDS2(config-profile)# port 3226
MDS2(config-profile)# interface fcip1
MDS2(config-if)# use-profile 1
MDS2(config-if)# peer-info ipaddr 198.18.69.9 port 3226
MDS2(config-if)# tcp-connections 1
MDS2(config-if)# switchport mode e
MDS2(config-if)# sw trunk mode on
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# passive-mode
MDS2(config-if)# no shut
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface fcip1
As mentioned the advanced features like SACK and PMTU are by default enabled. The best
command to verify the working of all your FCIP interfaces is the following command. It
shows you a summary of all important parameters of the interfaces.
MDS1(config-if)# show fcip summary
-------------------------------------------------------------------------------
Tun prof Eth-if peer-ip Status T W T Enc Comp Bandwidth rtt
E A A max/min (us)
-------------------------------------------------------------------------------
1 1 GE1/1.69 198.18.69.11 UP Y N N N N 1000M/500M 1000
MDS1(config-if)#
The detailed information for each FCIP tunnel is found in a normal show interface.
MDS1(config-if)# show interface fcip1
fcip1 is up (Trunking)
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:54:a9:40
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 1
Using Profile id 1 (interface GigabitEthernet1/1.69)
Peer Information
Peer Internet address is 198.18.69.11 and port is 3226
Write acceleration mode is configured off
Tape acceleration mode is configured off
Tape Accelerator flow control buffer size is automatic
FICON XRC Accelerator is configured off
Ficon Tape acceleration configured off for all vsans
IP Compression is disabled
Maximum number of TCP connections is 1
MDS1(config-if)#
Now multiple things are shown in this output. You can see the specifics of this particular FCIP
interface. This is separated in the information related to the TCP parameters of the session
and the Fibre Channel parameters of this interface. The FCIP interface is just a normal FC
interface when looked at from a Fabric perspective.
We can see in this output that we are indeed using 1 TCP connection and a non-default port,
which is initiated by MDS1, because it uses a destination port of 3226. When the switch
would’ve received the connection the source port would be 3226.
Now we have our first FCIP connection successfully up and running!
The next question in this task asks you to configure high availability for this FCIP
tunnel. We can solve this using 4 solutions.
The last and only correct option is the use of the Layer 3 redundancy protocol VRRP.
This protocol is the only FHRP protocol available on the MDS switches. With the use of
this protocol we can still have a single IP address for both the source interface and the
peer IP address. When something happens when the first GigabitEthernet interface
the VRRP configuration will issue a failover to the other. As the GigabitEthernet
interfaces on the MDS switches are just treated as Host interfaces, it’s perfectly
possible to run a local VRRP on the same MDS switch.
We will perform the VRRP configuration in a way that we do not need to change the FCIP
configuration as we are not allowed to do this according to the question. This means that we
need to re-use our interface IP addressing in the VRRP group configuration.
When we need to create additional IP addresses in the subnet this would be perfectly allowed
according to the current question requirements!
Let’s start by configuring the VRRP group using the correct IP addressing so we do not need to
change the FCIP interfaces for this change.
MDS1
MDS1(config-if)# interface gig1/1.69
MDS1(config-if)# ip address 198.18.69.99 255.255.255.0
MDS1(config-if)# vrrp 69
MDS1(config-if-vrrp)# address 198.18.69.9
MDS1(config-if-vrrp)# priority 120
MDS1(config-if-vrrp)# no shutdown
MDS1(config-if-vrrp)# exit
MDS2
MDS2(config-if)# interface gig1/1.69
MDS2(config-if)# ip address 198.18.69.111 255.255.255.0
MDS2(config-if)# vrrp 96
MDS2(config-if-vrrp)# address 198.18.69.11
MDS2(config-if-vrrp)# priority 120
MDS2(config-if-vrrp)# no shutdown
MDS2(config-if-vrrp)# exit
MDS2(config-if)# interface gig1/2.69
MDS2(config-if)# ip address 198.18.69.112 255.255.255.0
MDS2(config-if)# no shutdown
MDS2(config-if)# vrrp 96
MDS2(config-if-vrrp)# address 198.18.69.11
MDS2(config-if-vrrp)# no shutdown
MDS2(config-if-vrrp)# exit
Now with these VRRP groups configured we still have an active FCIP tunnel between the
originally configured endpoints as our endpoints didn’t change. We did have to configure
additional IP addresses, but there is no statement in the question that prohibits you from using
additional IP addresses in the subnet to accomplish the question.
MDS1
MDS1(config-if-vrrp)# sh vrrp
Interface VR IpVersion Pri Time Pre State VR IP addr
---------------------------------------------------------------------------
GigE1/1.69 69 IPv4 120 1 s master 198.18.69.9
GigE1/2.69 69 IPv4 100 1 s backup 198.18.69.9
MDS1(config-if-vrrp)#
MDS2
MDS2(config-if-vrrp)# sh vrrp
Interface VR IpVersion Pri Time Pre State VR IP addr
---------------------------------------------------------------------------
GigE1/1.69 96 IPv4 120 1 s master 198.18.69.11
GigE1/2.69 96 IPv4 100 1 s backup 198.18.69.11
MDS2(config-if-vrrp)#
Keep the VRRP group numbers different on the 2 MDS switches. You are configuring 2 VRRP
groups in the same VLAN. Remember that the group number is used in the virtual MAC
address of this virtual default gateway. When you configure equal group numbers
in the same VLAN, this will cause serious issues and an unworkable situation. The VRRP
configuration on both DMS switches should work fully independent of each other. Therefore
you will need to use different group numbers on both MDS switches.
We see that the local MDS switch is backup for itself. Therefore when something happens with
GigabitEthernet1/1 the second interface will become the master and the FCIP
connection remains up!
The next task will create the second FCIP connection between the MDS switches by using
another interface. Again we need to take care of a few specifics in this FCIP connection. When
you read ahead on the next number of questions you will see some specifics that need to be
applied to this connection.
For example, we need to ensure that this FCIP 2 connection receives a higher QoS priority.
Well we just saw in the verification of FCIP 1 that the default QoS marking is 0. So anything
higher than this is good for a better treatment in the IP network, when properly configured.
The second point is that the tunnel must be brought down when there are no packets received
for 90 seconds. This feature is the ‘tcp keepalive-timeout’ configured in the FCIP
profile, as it’s a TCP parameter.
MDS1
MDS1(config-if)# interface gig1/2.70
MDS1(config-if)# ip address 198.18.70.9 255.255.255.0
MDS1(config-if)# no shutdown
MDS1(config-if)#
MDS1(config-if)# fcip profile 2
MDS1(config-profile)# ip add 198.18.70.9
MDS1(config-profile)# tcp ?
cwm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisec
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes
MDS1(config-profile)# exit
MDS1(config)# interface fcip 2
MDS1(config-if)# use-profile 2
MDS1(config-if)# peer ip 198.18.70.11
MDS1(config-if)# qos ?
control Configure qos for control and data packets
We first create a new sub-interface on Gig1/2 with the new VLAN. We could have configured
this FCIP connection in the same VLAN as we are using different port numbers in the first
VLAN. So using the default port numbers would generate enough separation in the profile, as
discussed at the start of this task, to create an additional FCIP connection in the same VLAN.
In this case we create a new VLAN with different IP addressing. In the FCIP profile we
configure the keepalive-timeout to be 90 seconds, so the tunnel is brought down after
this time out expires.
Then under the FCIP interface we configure the QoS values for this interface. The requirement
is that it should be higher than the FCIP 1 interface. In this example you see that it’s possible
to create a different QoS mapping for both the Control session and the Data session.
Requirement for this to work is that you always use the 2 TCP connection configuration, which
is the default.
Finally we need to add this second FCIP connection toe the VSAN database as our Default
VSAN 1 is suspended, which means the interface will not come online when it is configured in
that VSAN.
Now we need to bring the connection up on the second MDS.
MDS2
MDS2(config-if)# interface gig1/2.70
MDS2(config-if)# ip address 198.18.70.11 255.255.255.0
MDS2(config-if)# no shutdown
MDS2(config-if)#
MDS2(config-if)# fcip profile 2
MDS2(config-profile)# ip add 198.18.70.11
MDS2(config-profile)# tcp keepalive-timeout 90
MDS2(config-profile)# exit
MDS2(config)# interface fcip 2
MDS2(config-if)# use-profile 2
MDS2(config-if)# peer ip 198.18.70.11
MDS2(config-if)# qos control 15 data 10
MDS2(config-if)# sw mod e
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# sw trunk allowed vsan add 50
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface fcip2
When both FCIP 1 and FCIP 2 are up and running we can configure the load balancing
parameters that we need to comply with.
We need to ensure that VSAN 10 traffic uses one link and VSAN 20 the other link. From the
standpoint of MDS2 this should be vice versa. This means that on MDS2 the traffic for VSAN 10
should use the FCIP 2 connection and VSAN 20 traffic should be using the FCIP 1
connection.
This means that we will be using the FSPF costs to alter the behavior of where traffic uses
which link. The FCIP interfaces are by default seen as 1Gbps links. We need to use these links
as the primary links for VSAN 10 and 20, as this is stated by the questions. Keep in mind to
lower the costs to ensure that this happens. As said the FCIP links are seen as 1Gbps links,
meaning they have a default FSPF cost of 1000.
MDS1
MDS1(config-if)# interface fcip1
MDS1(config-if)# fspf cost 10 vsan 10
MDS1(config-if)# fspf cost 20 vsan 20
MDS1(config-if)# interface fcip2
MDS1(config-if)# fspf cost 20 vsan 10
MDS1(config-if)# fspf cost 10 vsan 20
MDS2
MDS2(config-if)# interface fcip1
MDS2(config-if)# fspf cost 20 vsan 10
MDS2(config-if)# fspf cost 10 vsan 20
MDS2(config-if)# interface fcip2
MDS2(config-if)# fspf cost 10 vsan 10
MDS2(config-if)# fspf cost 20 vsan 20
With these changes the traffic will now flow across the correct FCIP links.
The next question in this task is regarding compression. FCIP connections can compress their
data to ensure a more optimal way of sending traffic across the links.
In the past there where 3 modes of compression. This has been changed to only 2 modes:
Auto mode
This is enabled by default and will compress files when necessary and when the other
side of the FCIP connection negotiates compression as well.
Mode 1
This mode is used for higher bandwidth links, so less compression. Don’t let yourself go
blind on the description you get from the CLI. As that will tell you that mode 1 is High
compression, but they actually mean it’s High Speed compression compared to Mode 2.
Mode 2
This mode enables a higher compression rate, but introduces latency due to the
compression algorithm that takes time to run.
As we want the highest compression as possible in this question we need to configure Mode 2
on both sides. Don’t let yourself be confused with the Mode 1 CLI comment!
MDS1
MDS1(config-if)# interface fcip2
MDS1(config-if)# ip-compression ?
<CR>
auto Auto compression setting
mode1 High compression
mode2 Moderate compression for medium bandwidth links
MDS2
MDS2(config-if)# interface fcip2
MDS2(config-if)# ip-compression mode2
MDS2(config-if)#
The compression modes need to match on both sides otherwise you could run into issues with
the FCIP tunnels. Auto mode will solve all these issues.
The next question in the task wants that FCIP 1 sends R_RDY messages locally. Which means
that the messages are sent by the local MDS where they should actually be sent by the remote
host. When a host receives a confirmation a packet is arrived it will close the write action on it’s
disk and assumes everything is fine. Now this responsibility comes in the hands of the MDS
switch.
If the MDS switches will send confirmations for write actions this means that the disks will
respond faster and will finish their actions faster.
This technology is called ‘write-accelerator’. It can be configured for both normals disks
and tapes. For tape traffic a little different style of acknowledgements is sent and received, so
this is adapted for by the MDS switch.
It’s easy to enable by using a single command. Ensure you have enabled this on both sides of
the connection, otherwise traffic could behave strangely.
MDS1
MDS1(config-if)# interface fcip1
MDS1(config-if)# write-accelerator
MDS2
MDS2(config-if)# interface fcip1
MDS2(config-if)# write-accelerator
The final question of this task asks to configure high availability for connection FCIP 2. We
should do this using a third FCIP connection and we can not use Ethernet port-channels for the
high availability.
This leaves only one option which is configuring a Fibre Channel based port-channel
configuration with FCIP connections.
We need to ensure that the tasks that we configured before this will be kept in the port-
channel configuration so we don’t loose any points there.
The task mentions that we should keep high availability in mind when we are configuring this
question. To ensure we have the most resiliency on this third FCIP connection we should use a
different Ethernet interface as well, which we can easily do.
We can use the GigabitEthernet1/1 connection in VLAN 70, as we haven’t configured that
interface yet. We will need to use different IP addressing in this case as we share the same
VLAN. The question does not prohibit the creation of additional IP addresses in the subnet, so
we can do this.
Because we are creating a Fibre Channel port-channel the FSPF protocol does not notice the
change when one interface is lost in the port-channel FSPF costs will need to remain the same
for the configured VSANs in the previous question.
MDS1
MDS1(config-if)# interface gig1/1.70
MDS1(config-if)# ip address 198.18.70.99 255.255.255.0
MDS1(config-if)# no shutdown
MDS1(config-if)#
MDS1(config-if)# fcip profile 3
MDS1(config-profile)# ip add 198.18.70.99
MDS1(config-profile)# tcp keepalive-timeout 90
MDS1(config-profile)# exit
MDS1(config)# interface fcip 3
MDS1(config-if)# use-profile 3
MDS1(config-if)# peer ip 198.18.70.111
MDS1(config-if)# qos control 15 data 10
MDS1(config-if)# sw mod e
MDS1(config-if)# sw trunk allowed vsan 10
MDS1(config-if)# sw trunk allowed vsan add 20
MDS1(config-if)# sw trunk allowed vsan add 50
MDS1(config-if)#
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface fcip3
MDS2
MDS2(config-if)# interface gig1/1.70
MDS2(config-if)# ip address 198.18.70.111 255.255.255.0
MDS2(config-if)# no shutdown
MDS2(config-if)#
MDS2(config-if)# fcip profile 3
MDS2(config-profile)# ip add 198.18.70.111
MDS2(config-profile)# tcp keepalive-timeout 90
MDS2(config-profile)# exit
MDS2(config)# interface fcip 3
MDS2(config-if)# use-profile 3
MDS2(config-if)# peer ip 198.18.70.99
MDS2(config-if)# qos control 15 data 10
MDS2(config-if)# sw mod e
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# sw trunk allowed vsan add 50
MDS2(config-if)#
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface fcip3
Now we configured the third FCIP connection across the other GigabitEthernet interface
so we have high availability for those 2 FCIP connections.
To finish we need to force the tunnels to form a port-channel. We need to ensure we keep
the configuration of FCIP 2 leading for the port-channel.
MDS1
MDS1(config-if)# interface fcip2
MDS1(config-if)# channel-group 99
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fcip3
MDS1(config-if)# channel-group 99
MDS1(config-if)# no shutdown
MDS1(config-if)#
MDS1(config-if)# interface port-channel 99
MDS1(config-if)# sw mod e
MDS1(config-if)# sw trunk allowed vsan 10
MDS1(config-if)# sw trunk allowed vsan add 20
MDS1(config-if)# sw trunk allowed vsan add 50
MDS1(config-if)# ip-compression mode2
MDS1(config-if)# fspf cost 20 vsan 10
MDS1(config-if)# fspf cost 10 vsan 20
MDS1(config-if)# no shutdown
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface port-channel99
MDS2
MDS2(config-if)# interface fcip2
MDS2(config-if)# channel-group 99
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fcip3
MDS2(config-if)# channel-group 99
MDS2(config-if)# no shutdown
MDS2(config-if)#
MDS2(config-if)# interface port-channel 99
MDS2(config-if)# sw mod e
MDS2(config-if)# sw trunk allowed vsan 10
MDS2(config-if)# sw trunk allowed vsan add 20
MDS2(config-if)# sw trunk allowed vsan add 50
MDS2(config-if)# ip-compression mode2
MDS2(config-if)# fspf cost 10 vsan 10
MDS2(config-if)# fspf cost 20 vsan 20
MDS2(config-if)# no shutdown
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface port-channel99
Now the port-channel has formed and we have 2 members configured in this port-channel. It
transports all the VSANs that FCIP 2 does and it has the same interface properties like FSPF
costs for the VSANs and we have configured IP-Compression like we did on FCIP 2.
Finally don’t forget to add this connection to the VSAN database as well, otherwise it will
remain down.
We now have a high available FC port-channel connection for FCIP!
MDS1(config-if-vrrp)# authentication ?
md5 Set the MD5 authentication key (16 char max)
text Set the authentication password (8 char max)
One thing different about configuration of authentication in VRRP on the MDS switches is that
you need to configure a Security Parameter Index (SPI). This is a unique key for the
authentication key that is communicated to the neighbor. This SPI should be unique for each
VSAN!
MDS2
MDS2(config-if)# interface gig1/1.69
MDS2(config-if)# vrrp 96
MDS2(config-if-vrrp)# shut
MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1
MDS2(config-if-vrrp)# no shut
MDS2(config-if-vrrp)#
MDS2(config-if)# interface gig1/2.69
MDS2(config-if)# vrrp 96
MDS2(config-if-vrrp)# shut
MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1
MDS2(config-if-vrrp)# no shut
MDS2(config-if-vrrp)#
After this configuration the communication of the VRRP is now authenticated between the
neighbors.
The next 2 tasks are regarding the security of the FCIP 1 connection. This configuration is
quite complex on the MDS switch and you should pay close attention to what you configure
here.
The IPsec feature is not available on all platforms, but it is available on the platform in the CCIE
Data Center lab topology, MDS 9222i. It’s not available on the IPS-4 and IPS-8 line cards.
You also require a ENTERPRISE license for this feature.
Configuration is similar to the IOS configuration for IPsec VPNs, but you will find a little
different commands are used here and there.
The question states that we should secure the FCIP 1 connection. Keep in mind that you use
the VRRP configuration as the peer address configurations, but you should apply the IPsec
tunnels to both interfaces where the tunnel could terminate. When this isn’t done, you will
loose points as the FCIP 1 connection is high available and your security not!
Let’s start the configuration on MDS1.
MDS1
MDS1(config)# feature crypto ike
MDS1(config)# feature crypto ipsec
MDS1(config)#
MDS1(config)# crypto ike domain ipsec
MDS1(config-ike-ipsec)# identity address
MDS1(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.11
MDS1(config-ike-ipsec)# policy 1
MDS1(config-ike-ipsec-policy)# encr aes
MDS1(config-ike-ipsec-policy)# hash md5
MDS1(config-ike-ipsec-policy)# authentication pre-share
MDS1(config-ike-ipsec-policy)# group 2
MDS1(config-ike-ipsec-policy)# exit
MDS1(config-ike-ipsec)#
MDS1(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.9 0.0.0.0
198.18.69.11 0.0.0.0
MDS1(config)# crypto transform-set domain ipsec FCIP1 esp-aes ?
128 ESP transform using AES 128 bit cipher
256 ESP transform using AES 256 bit cipher
We configured the IPsec tunnel on both interfaces for only the traffic that is using the VRRP
group addresses. Other traffic is not hit by the configured access-list and will therefore not
hit the crypto map and traffic will not be encrypted. Only the IP addresses that are
configured in the access-list will hit the crypto map and will be encrypted.
Now we need to finish the configuration for the second MDS.
MDS2
MDS2(config)# feature crypto ike
MDS2(config)# feature crypto ipsec
MDS2(config)#
MDS2(config)# crypto ike domain ipsec
MDS2(config-ike-ipsec)# identity address
MDS2(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.9
MDS2(config-ike-ipsec)# policy 1
MDS2(config-ike-ipsec-policy)# encr aes
MDS2(config-ike-ipsec-policy)# hash md5
MDS2(config-ike-ipsec-policy)# authentication pre-share
MDS2(config-ike-ipsec-policy)# group 2
MDS2(config-ike-ipsec-policy)# exit
MDS2(config-ike-ipsec)#
MDS2(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.11 0.0.0.0
198.18.69.9 0.0.0.0
MDS2(config)# crypto transform-set domain ipsec FCIP1 esp-aes 128 esp-md5-hmac
MDS2(config)# crypto map domain ipsec FCIP 1
MDS2(config-crypto-map-ip)# match address FCIP1IPSEC
MDS2(config-crypto-map-ip)# set peer 198.18.69.11
MDS2(config-crypto-map-ip)# set transform-set FCIP1
MDS2(config-crypto-map-ip)# interface gig1/1.69
MDS2(config-if)# crypto map domain ipsec FCIP
MDS2(config-if)# interface gig1/2.69
MDS2(config-if)# crypto map domain ipsec FCIP
Now all FCIP 1 related traffic is secured and we have finished task 3!
The virtual initiator and virtual target are only able to communicate to
each other and not to other hosts or targets in the Fibre Channel fabric
Use a different GigabitEthernet interface than the FCIP interface
As you are using this feature to test the ethernet interface that is used for the FCIP
connection, you should not use the same GigabitEthernet interface as the tunnel
itself because it impacts the amount of traffic in the link
This interface does not need to be up, it is just used to generate the traffic on
iSCSI needs to be enabled
Be aware that you require zoning changes before the hosts are able to communicate to each
other. As we have no zoning configured for VSAN 50, which is the VSAN we should use in
this task. We can just allow traffic to be sent in this VSAN in the Default Zone.
We require to enable a couple of things before we start the san-extension-tuner
configuration.
MDS1
MDS1(config)# feature san-ext-tuner
MDS1(config)# feature iscsi
MDS1(config)# int gig1/3
MDS1(config-if)# no shut
MDS1(config-if)# int iscsi1/3
^
Invalid range at '^' marker.
MDS1(config)# iscsi enable module 1
MDS1(config)# int iscsi1/3
MDS1(config-if)# no shut
We also need to enable the iSCSI feature for this task. When you enable iSCSI you also need
to enable the iSCSI interface. The interface numbering is equal to the GigabitEthernet
number. Before this interface exists, you also need to enable the iSCSI feature on the line
card (module) that you want to enable the iSCSI interface on. In the case of a fixed chassis,
like the MDS 9222i, we only have 1 module.
After it’s enabled we can enable the san-extension-tuner host in the network and we need
to enable the Default Zone to allow traffic for being sent between hosts.
MDS1
MDS1(config)# zone default-zone permit vsan 50
MDS1(config)# exit
MDS1(config)# san-ext-tuner
^
% Incomplete command at '^' marker.
MDS1(san-ext)# nwwn ?
<hh:hh:hh:hh:hh:hh:hh:hh> Enter the N-WWN of san extension tuner
After the N-port is configured you can see a successful FLOGI being done in the configured
VSAN with the nWWN and pWWN that you just configured.
MDS1(san-ext-nport)# sh flogi data v 50
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
iscsi1/3 50 0x6c0000 10:00:00:00:00:00:00:02 10:00:00:00:00:00:00:01
MDS1(san-ext-nport)#
Now we have 2 virtual hosts in VSAN 50 that we can use to create our data
communication flows on for our san-extension-tuner feature.
Do not forget to enable the continuous keyword at the end of the configuration. The
question clearly states that you should start a continuous flow of traffic
We currently have traffic in one direction. The task states that we should enable the read
commands in both directions. So we need to configure the same configuration on the other
MDS switch as well.
MDS2
MDS2# san-ext-tuner
MDS2(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface gig1/3
MDS2(san-ext-nport)# read command-id 1 target 20:00:00:00:00:00:00:02 transfer-size
512000 outstanding-ios 2 continuous
After this configuration is applied the read command immediately starts and traffic is flowing
across the FCIP 2 connection, because this is the only FCIP connection where VSAN 50 is
allowed.
Task 5: iSCSI
The next task is all about the iSCSI feature on the Cisco MDS platform. This feature has been
available since a long time and is still available, however a more newer version is also available,
which is discussed in the next task.
Using the iSCSI feature on the MDS platform you are able to extend IP connected hosts
(initiators) on to the Fibre Channel network. This means that the Fibre Channel fabric will have
an entry in the name-server for this IP host just like if it was a Fibre Channel connected
server.
On the other side, you can create iSCSI targets based on Fibre Channel targets. You can
take any target in the Fibre Channel fabric and make this accessible through iSCSI.
The iSCSI feature is available on the same IP interfaces as the FCIP feature is available.
The MDS switch makes the conversion between the IP traffic and Fibre Channel traffic. Hosts
that log-in using iSCSI do not know they are actually communicating with a Fibre Channel
switch and a Fibre Channel disk. The iSCSI header is stripped and a Fibre Channel header is
placed on top of it. Of course the return traffic works exactly vice versa.
The drawing below illustrates the working of the iSCSI portal on the MDS switch.
The most easy option for configuring iSCSI is to use the dynamic mapping feature.
With this feature it’s possible to present every target available on the Fibre Channel network as
an iSCSI target. As the first question states in this task this is prohibited!
This leaves you with the option to configure static targets. Here you have options to
configure targets either on a target basis or you can even specify which LUN can be accesses
through the iSCSI network.
As with targets you can also dynamically create virtual initiators. Meaning that every
device that tries a log-in on the MDS switch IP address on the iSCSI portal, will
automatically get a virtual initiator assigned. Again we do not want this and we will be
configuring static initiators so we have full control over which host can access which
target, as the initiators will need to be included in zoning!
The 2 options that you can use to define a iSCSI initiator are:
iSCSI Qualified Name (iQN)
This IQN is a clear-text name that the iSCSI device uses to identify itself. When an
initiator has multiple IP addresses it will still be just a single Virtual N-Port in the
Fibre Channel fabric
IP address
The Virtual N-port can also be created according to the IP address that the iSCSI
initiator uses to access the MDS. Each IP address is another virtual N-port and
another host in the Fibre Channel fabric
In the initial configuration we will be using the Transparent Initiator mode, which
means that every iSCSI initiator will appear as an Initiator in the FC fabric network
(Virtual N-port).
During this task we will also be configuring the other mode, which is the Proxy Initiator
mode. With the use of this mode. The MDS switch will act as a single virtual N-port and all
iSCSI initiators are hidden behind this ‘proxy’ feature. This means that you can not use
zoning to determine which iSCSI initiator can access which target.
However there are multiple ways in the iSCSI configuration where you can specify which
target can access which initiator.
Now we start the configuration of our first couple questions in this iSCSI task.
We call an iSCSI interface on the MDS switch a iSCSI Portal, as it’s a single portal where
you can log-in and access all targets that you have the right to.
We start with some infrastructure configuration to create the iSCSI portal.
The iSCSI interface that will be configured here is a virtual interface that is an overlay on
top of the GigabitEthernet interface. The numbering of the iSCSI interface is equal to the
GigabitEthernet numbering. This interface is required to be enabled, to really activate the
iSCSI portal on the MDS. Besides that, which we already did in the previous task, is that we
need to enable the iSCSI feature globally and per line-card where the feature needs to be
enabled.
We configure the VLAN according to Drawing 2. There we find that we should be using VLAN
71. Therefore we need to create another sub-interface for the VLAN. It’s not possible on the
Cisco MDS switch to configure the iSCSI portal on a sub-interface. Meaning that all iSCSI
interface configuration that you do will be used on all sub-interfaces that are configured.
This means that you can not create different iSCSI portals on different sub-interfaces, but
only a single portal per physical interface.
MDS1
MDS1(config)# feature iscsi
MDS1(config)# iscsi enable module 1
MDS1(config)# interface gig1/1
MDS1(config-if)# no shut
MDS1(config-if)# int gi1/1.70
MDS1(config-if)# ip add 198.18.70.9 255.255.255.0
MDS1(config-if)# no shut
Like previously mentioned. You can not create an iSCSI sub-interface only a global interface.
MDS1(config-if)# int iscsi1/1
MDS1(config-if)# port 3261
MDS1(config-if)# tcp ?
cwm Enable congestion window monitoring
keepalive-timeout Set keep alive timeout in sec
max-bandwidth-kbps Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps Configure maximum bandwidth
max-jitter Configure jitter for iSCSI in micro seconds
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisec
pmtu-enable Enable PMTU Discovery
sack-enable Enable SACK option for TCP
send-buffer-size Send buffer size in KBytes
The TCP parameters that can be configured are similar to the options that are available for
configuring FCIP connections.
MDS1(config-if)# qos ?
<0-63> QOS code point value
MDS1(config-if)# qos 22
MDS1(config-if)# no shut
MDS1(config-if)# vsan data
MDS1(config-vsan-db)# vsan 10 interface iscsi1/1
Finally the last configuration that needs to be done is by putting the iSCSI interface into
another VSAN, because the Default VSAN is suspended. When this is not configured, no
matter what kind of VSAN configuration is done under the virtual initiators or virtual
targets the interface will remain down, because the Default VSAN is suspended.
The different topics in this infrastructure task is that the portal should use a non-default
port. By default iSCSI uses TCP 3260 for all communications. We now change this to TCP
3261. The configuration is done under the iSCSI interface. Under the iSCSI interface the
characteristics of the traffic, like TCP parameters, port number, etc.
MDS1(config-if)# do sh int iscsi1/1
iscsi1/1 is up
Hardware is GigabitEthernet
Port WWN is 20:13:00:0d:ec:54:a9:80
Admin port mode is ISCSI
snmp link state traps are enabled
Port mode is ISCSI
Port vsan is 10
Speed is 1 Gbps
Interface last changed at Tue Oct 23 14:13:46 2012
Now our iSCSI portal is up and running, but since we are using no dynamic targets or
initiators nothing will be able to log-in to it yet.
As shown in this output, you can see that by default initiators are recognized by their name
(iQN). This needs to be changed for this portal, as we want to use the IP address of the
initiator as we look at the following question. This can be changed per iSCSI interface and
therefore per iSCSI portal configured on the MDS switch.
MDS1
MDS1(config)# int iscsi1/1
MDS1(config-if)# switchport ?
description Enter description of maximum 80 characters
initiator Configure iSCSI initiator id mode
proxy-initiator Configure iSCSI proxy initiator mode
timestamp-check Enable timestamp check
The Cisco MDS switch allows you to configure both manual nWWN and pWWN’s to the initiators
and you can chose for the feature that the MDS will select pWWNs or a nWWN automatically
based on a pool.
In this case we will use statically configured WWNs.
MDS1
MDS1(config)# iscsi initiator ip-address 198.18.71.100
MDS1(config-iscsi-init)# ?
mutual-chap Assign username for initiator's challenge (mutual chap)
no Negate a command or set its defaults
static Create static nWWN or pWWN for the initiator
username Assign username for iSCSI login authentication
vsan Assign VSAN membership for the initiator
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
We need to ensure that the initiator is participating in VSAN 20, but we can not configure
this under the initiator by using the vsan command in this case. With the use of this command
you are able to configure different initiators in different VSANs. This creates a lot of
flexibility in the deployment of the initiators.
We just configured the iSCSI interface under the VSAN database. So we need to change this to
VSAN 20 to support our question and let the initiator have access to resources in VSAN 20.
MDS1
MDS1(config-if)# vsan data
MDS1(config-vsan-db)# vsan 20 interface iscsi1/1
Now that we have finished our configuration of our initiator we can verify it’s configuration.
Don’t get scared when you issue the first command and you don’t see any results.
This has to do with the fact that the show iscsi initiator command will only show you
the currently active iSCSI sessions. With the configured keyword behind it, it will show you
the configuration of your iSCSI initiators.
You will also not see any FLOGI’s or Name-Server entries until the iSCSI session is active
and working.
MDS1
MDS1(config-vsan-db)# sh iscsi initiator
MDS1(config-vsan-db)#
Now that the initiator part is configured we can start with configuring targets for this
iSCSI host. The host should be able to access the JBOD 2 Disk 1 in VSAN 20.
The next question states that you should be configuring a virtual target for the JBOD 2
Disk 1. This is done using a similar configuration as the initiator.
Before the virtual target can be accessed you need to configure which initiators are
allowed to access it. This means, the initiators that will see it when they discover the
targets through iSCSI. After this configuration you still need zoning where you need to give
the virtual pWWN or IP-address access to the Fibre Channel target.
Next is that you can configure the virtual-target to have it showing up on which iSCSI
portal. This is done by specifying the GigabitEthernet interface (and with that the
portal) where it’s available.
MDS1(config-iscsi-tgt)# initiator ?
WORD Enter iSCSI initiator name (Max Size 223)
ip Allow iSCSI initiator access to this target by ip address
You can specify an entire range of IP addresses with this command, when you do not configure
any netmask, it means that you only allow a single host.
MDS1
MDS1(config-iscsi-tgt)# advertise ?
interface Interface eth-interface-to-advertise
Within the current iSCSI configuration it’s only possible to configure a pWWN of the target you
want to make available. You will see in the next task that there are more options available, like
using the device-alias instead of the pWWN, which is a much more flexible way.
MDS1
MDS1(config-iscsi-tgt)# pwwn 21:00:00:11:0d:18:fe:ce ?
<CR>
fc-lun FC lun
secondary-pWWN Enter the secondary pWWN of the fc-target
To finish up, we need to configure zoning so the host as access to the target. This can be
based on multiple things.
MDS1
MDS1(config)# zone name iSCSI_1 vsan 20
MDS1(config-zone)# member ?
device-alias Add device-alias member to zone
domain-id Add member based on domain-id,port-number
fcalias Add fcalias to zone
fcid Add FCID member to zone
fwwn Add Fabric Port WWN member to zone
interface Add member based on interface
ip-address Add IP address member to zone
pwwn Add Port WWN member to zone
symbolic-nodename Add Symbolic Node Name member to zone
You can use the following options in letting iSCSI initiators access Fibre Channel
targets:
IP address
Just use the IP address of the initiator, just like we configured the virtual
initiator
IQN
pWWN
You can use the virtual pWWN that is manually assigned by you or system assigned by
the MDS switch to give the initiator access to the FC Disk. This is also a very flexible
option, but it makes you zoning configuration harder to read
MDS1
The next question requires you to configure authentication. It’s possible to authenticate your
iSCSI session in both directions, which is what we are looking for.
You can authenticate the iSCSI users according to the local database, which is what we are
going to use here, or through an RADIUS or TACACS+ server.
Be aware that when you are configuring the user, to not forget the special ‘iscsi’ keyword
behind the line, otherwise it will definitely not work!
Especially when you are configuring mutual-chap, where the second authentication is
configured right under the virtual initiator, you might think that it enough, but you are
not done there. You need to configure the initial CHAP username/password as well!
MDS1
MDS1(config)# iscsi initiator ip-address 198.18.71.100
MDS1(config-iscsi-init)# username iSCSI1
MDS1(config-iscsi-init)# mutual-chap ?
username Assign username for initiator's challenge (mutual chap)
After this the configuration of this virtual target is done and we can verify our
configuration.
MDS1(config)# show iscsi virtual-target
target: iqn.iscsi-disk-jbod2-disk1
Port WWN 21:00:00:11:0d:18:fe:ce
Configured node (iSCSI)
No. of advertised interface: 1
GigabitEthernet 1/1.71
No. of initiators permitted: 1
initiator 198.18.71.100/32 is permitted
All initiator permit is disabled
Trespass support is disabled
MDS1(config)#
All configuration parameters that we configured are available. Now we check our changed
initiator, where we added the authentication.
MDS1(config)# show iscsi initiator configured
iSCSI Node name is 198.18.71.100
MDS1(config)#
The authentication is added and we see that the same username is used for both the login and
the second CHAP (mutual).
The next question asks you to configure another virtual target with a LUN specified. The
iSCSI feature supports LUN masking, so you are able to convert the LUN IDs of targets to
another LUN ID on the iSCSI level.
The question will also be using another great feature of the iSCSI feature of the Cisco MDS
platform. The iSCSI virtual-target feature can take care of high availability. Where it will
be using a sort of SDV (SAN Device Virtualization) technology to ensure this. The
iSCSI virtual target will have access to a primary pWWN and a secondary pWWN that it can
use to failover to as soon as the primary fails.
Configuration can generate quite a long command!
MDS1
MDS1(config)# iscsi virtual-target name iqn.iscsi-disk-jbod1-disk3
MDS1(config-iscsi-tgt)# sh device-alias data
device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce
device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce
device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce
device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce
device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce
device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce
device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce
device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce
Here we can see that we can even configure a different Fibre Channel LUN for both the primary
and the backup target. As the question only specified LUN 10, we will continue to use this on
the backup target as well.
MDS1
MDS1(config-iscsi-tgt)# all-initiator-permit
As the question stated that ‘iSCSI initiators’ should have access to the virtual-
target, this indicates that we want to allow all initiators that will be logging in, to have
access to this specific target. As we only have 1 initiator configured, this will be currently
the only one that has access to it.
MDS1
MDS1(config-iscsi-tgt)# revert-primary-port
The question also stated that the backup disk should fall back to the primary when this is
available again after repair.
MDS1
MDS1(config-iscsi-tgt)# trespass
MDS1(config-iscsi-tgt)#
The trespass support that is asked for, is currently a hidden command, but still available in
the configuration and a supported feature. This feature ensures that Logical Units are
transferred over to the failover target.
Do not forget to enable zoning for the initiator that we configured! In this zoning it’s
very important not to forget to add both the disks that you are using, so both the primary and
the backup target should be mentioned in the zone, before the initiator has access to it.
MDS1
MDS1(config)# zone name iSCSI_2 vsan 20
MDS1(config-zone)# member ip-address 198.18.71.100
MDS1(config-zone)# member device-alias J1D3
MDS1(config-zone)# member device-alias J2D3
MDS1(config-zone)# zoneset name ZS1 vsan 20
MDS1(config-zoneset)# member iSCSI_2
MDS1(config-zoneset)# zoneset activate name ZS1 vsan 20
MDS1(config)# zone commit vsan 20
Zoneset activation initiated. check zone status
MDS1(config)#
target: iqn.iscsi-disk-jbod1-disk3
Port WWN c6:83:00:11:0d:18:fc:ce
Secondary PWWN 21:00:00:11:0d:18:a0:ce
Configured node (iSCSI)
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x000a, Secondary FC LUN: 0x000a
All initiator permit is enabled
Trespass support is enabled
Revert to primary support is enabled
MDS1(config-iscsi-tgt)#
In the second virtual target you will find the 2 pWWNs of the disks and the features that are all
enabled.
The final feature that should be enabled on this iSCSI portal is that we want you to improve
the read performance on the switch.
This is related to the way the iSCSI packets are treated in the MDS switch. The MDS can treat
iSCSI frames just like an Ethernet switch has different modes to transfer Ethernet packets.
The downside of this mode is that it lowers the total amount of data transfer that is
possible. However the positive side is that you get lower latency and you are able to use
the data-digest feature.
Store-and-Forward
This is the default and recommended mode. Data-digest can not be used, but you
do get a faster data transfer performance in the iSCSI process. The entire iSCSI
packet is cached inside the box before the next action is taken.
Cut-Thru
MDS1(config-if)# mode ?
store-and-forward Enable store-and-forward
The different modes are not available through the CLI context help, but they are available. You
can verify using the following output.
MDS1(config-if)# do sh int iscsi1/1
iscsi1/1 is down (Initializing)
Hardware is GigabitEthernet
Port WWN is 20:13:00:05:9b:70:5a:00
Admin port mode is ISCSI
snmp link state traps are enabled
Port vsan is 20
iSCSI initiator is identified by ip address
Number of iSCSI session: 0 (discovery session: 0)
Number of TCP connection: 0
Configured TCP parameters
Local Port is 3261
PMTU discover is enabled, reset timeout is 3600 sec
Keepalive-timeout is 60 sec
Minimum-retransmit-time is 300 ms
Max-retransmissions 4
Sack is enabled
QOS code point is 0
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 70000 kbps
Configured round trip time is 1000 usec
Send buffer size is 4096 KB
Congestion window monitoring is enabled, burst size is 50 KB
Configured maximum jitter is 500 us
Forwarding mode: cut-thru
TMF Queuing Mode: enabled
Proxy Initiator Mode: disabled
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
iSCSI statistics
Input 0 packets, 0 bytes
Command 0 pdus, Data-out 0 pdus, 0 bytes, Unsolicited 0 bytes
Output 0 packets, 0 bytes
Response 0 pdus (with sense 0), R2T 0 pdus
Data-in 0 pdus, 0 bytes
MDS1(config-if)#
Next we are configuring a new iSCSI portal on the other MDS switch.
This new portal is configured in the same iSCSI VLAN as we used in the previous questions,
but as we are using another IP address on another MDS switch, there is no problem in any
overlap.
This portal will work a little different than the previous one as the iSCSI initiators should
all be hidden behind a single nWWN and pWWN. This is called the ‘proxy-initiator-mode’.
With this mode the MDS switch will act as the Fibre Channel host in the network where all
traffic is sent and the switch will convert it then to iSCSI packets destined to the correct
initiator.
This mode introduces flexibility as you only have a single entry in your FC Name-Server for the
iSCSI hosts. It hides the iSCSI hosts from the network, but it eliminates the fact that you can
zone the targets and initiators together. It also creates simplicity as you have most
security features to secure the combination of initiators and targets in the virtual-target
configuration as you would have in zoning. Therefore the adding of each initiator and
each target (including possible backup targets) will create a lot of additional zones that
need to be configured.
Within the proxy-initiator-mode, the MDS will create a virtual N-port per iSCSI
portal (per iSCSI interface to be exact). So when you enable this mode on multiple
iSCSI interfaces, it will create additional N-ports in the FC Fabric, which you all need to
add to the zones.
The following drawing illustrates how the Proxy Initiator Mode works.
We now configure the iSCSI portal on MDS2. Most configuration is equal to the other iSCSI
portal on MDS1, except that the behavior is very different.
Keep in mind that when you configure proxy-initiator-mode, it’s not possible anymore
to do your zoning based on IQNs or IP addresses.
MDS2
MDS2(config)# feature iscsi
MDS2(config)# iscsi enable mod 1
MDS2(config)# int gig1/1.71
MDS2(config-if)# ip add 198.18.71.11 255.255.255.0
MDS2(config-if)# no shut
MDS2(config-if)# int gig1/1
MDS2(config-if)# no shut
MDS2(config-if)# int iscsi1/1
MDS2(config-if)# no shut
We are first required to enable iSCSI on this switch as it wasn’t enabled before.
MDS2
MDS2(config-if)# switchport proxy-initiator ?
<CR>
nWWN Configure nWWN for proxy initiator mode
In this example we are using manually configured nWWN and pWWN for the proxy host that is
generated on the switch. This makes it easier to define our zoning later on as we do not need
to look up the values, but we configured them by ourselves. Be aware that this should be a
globally unique value. In the CCIE Data Center lab exam you do not have that many Fibre
Channel connected devices, so it should be fine to use these kinds of values for lab purposes.
Now we can verify our proxy-initiator mode configuration.
MDS2(config-if)# sh int iscsi1/1
iscsi1/1 is up
Hardware is GigabitEthernet
Port WWN is 20:13:00:05:9b:77:0d:c0
Admin port mode is ISCSI
snmp link state traps are enabled
Port mode is ISCSI
Port vsan is 1
Speed is 1 Gbps
Interface last changed at Wed Oct 24 23:43:44 2012
The next question requires you to configure the data-digest feature. The funny thing is
that this is a feature that can not be directly configured. However, as just discussed, this feature
is only available in a certain forwarding mode.
When enabling this feature the data-digest feature is automatically enabled when the
iSCSI initiators send requests for this!
MDS2
MDS2(config-if)# int iscsi1/1
MDS2(config-if)# mode pass-thru
Now we are going to configure 3 initiators that will be logging in to the iSCSI portal.
They are member of VSAN 10, but we can’t configure the VSAN database for this VSAN.
Remember that the iSCSI portal is currently in the suspended Default VSAN and will
not come online!
MDS2
MDS2(config)# iscsi initiator name iqn.initiator-server-1
MDS2(config-iscsi-init)# vsan 10
MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-2
MDS2(config-iscsi-init)# vsan 10
MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-3
MDS2(config-iscsi-init)# vsan 10
We can configure initiators to be in any VSAN. This also means that the same iSCSI portal
can be using resources from different VSANs. In each VSAN the switch will create a virtual
N-port when necessary.
MDS2
MDS2(config-iscsi-init)# vsan data
MDS2(config-vsan-db)# vsan 20 interface iscsi1/1
To overcome the limitation that we can not configure VSAN 10 in the VSAN database
configuration. We just enable the iSCSI portal interface in another VSAN. This has no impact
on the working of the initiators. As the configuration under the initiator configuration
is leading over the VSAN database configuration.
Next we need to give the initiators access to a virtual target.
MDS2
MDS2(config-iscsi-tgt)# sh device-alias database
device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce
device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce
device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce
device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce
device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce
device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce
device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce
device-alias name J2D4 pwwn 21:00:00:11:0d:18:a1:ce
After configuring the virtual-target we can verify our target and initiator
configuration.
MDS2(config-iscsi-tgt)# sh iscsi virtual-target
target: iqn.jbod-disk-j1d1
Port WWN c6:83:00:11:0d:18:fa:ce
Configured node (iSCSI)
No. of initiators permitted: 3
initiator iqn.initiator-server-1 is permitted
initiator iqn.initiator-server-2 is permitted
initiator iqn.initiator-server-3 is permitted
All initiator permit is disabled
Trespass support is disabled
Revert to primary support is disabled
MDS2(config-iscsi-tgt)#
MDS2(config-iscsi-tgt)#
We can see that the target only allows certain initiators to connect to it and that it is
connected to the pWWN of J1D1.
MDS2(config-iscsi-tgt)#
MDS2(config-iscsi-tgt)# show iscsi initiator configured
iSCSI Node name is iqn.initiator-server-1
Member of vsans: 10
Configured node (iSCSI)
Member of vsans: 10
Configured node (iSCSI)
MDS2(config-iscsi-tgt)#
The initiators are configured so that they are members of VSAN 10.
Finally we need to enable zoning for these initiators and a target. Where we can only
use 2 entries in the zoning.
This is relatively easy as we are using the proxy initiator mode, so we can use the pWWN
of the proxy host and the device-alias of the J1D1 disk and we have our zoning
completed.
MDS2(config-iscsi-tgt)# zone name ISCSI1 vsan 20
MDS2(config-zone)# zone name ISCSI1 vsan 10
MDS2(config-zone)# member device-alias J1D1
MDS2(config-zone)# member pwwn aa:bb:cc:dd:ee:ff:00:11
MDS2(config-zone)# zoneset name ZS1 vsan 10
MDS2(config-zoneset)# member ISCSI1
MDS2(config-zoneset)# zoneset activate name ZS1 vsan 10
Zoneset activation initiated. check zone status
MDS2(config)#
With this configuration we have our iSCSI configuration completed. In the next task we will be
configuring the next version of the iSCSI feature on the MDS platform called iSLB!
Task 6: iSLB
This final task of this chapter will be about configuring iSLB. This feature is the next version of
iSCSI with more features and has more features specifically related to high availability and
load balancing between different MDS switches.
The iSLB feature has a couple unique features that are missing from the iSCSI
implementation, which might make it hard to roll-out this feature for hundreds of iSCSI
initiators.
By the use of auto-zoning you are able to automatically zone the initiator and
virtual-target together without having to manually configure the zoning. The
combination of the initiator and virtual-target is done by the use of so-called
‘initiator targets’. This means that under the initiator configuration the
virtual-target configuration is done.
By using CFS distribution for this feature you can push the configuration of the
iSLB feature to multiple MDS switches in the fabric. This eases the configuration and
ensures that you have equal configurations for initiators and targets across the
whole fabric.
Load balancing of initiator requests across MDS switches using VRRP and iSCSI redirect
When you are using VRRP, there is a single switch that will receive all iSCSI login
requests. The iSLB feature implements iSCSI login redirection so the login request is
redirected to the other MDS switch which is also participating in the VRRP configuration.
Keep in mind that you need to keep iSCSI enabled on the switch as well. This is required when
you are using more than 200 iSLB initiators, but it’s always a best practice to follow.
iSLB can be using dynamic initiators and targets just like iSCSI. By default the auto
learning is using the iSCSI feature, but you can change this to use the iSLB feature instead.
When the initiators or targets are already learned through the iSCSI feature you are
not able to convert these to iSLB variants. You need to remove them and re-discover the
initiators and targets.
Again in our task we are not allowed to use any dynamic learning, just like for iSCSI.
The initial questions in this task are related to building up the iSLB portal on a different
interface than were we configured the iSCSI portal.
When we read the first 3 questions we have enough information to start building our portal.
We can only configure MDS2 for configuration changes to initiators and targets. Which
means that we should be using distribution of configuration using the CFS protocol.
The third question indicates that the portal should be high available. Where the only option
we have is to use the VRRP.
Let’s start the configuration on our switches:
MDS1
MDS1(config)# feature iscsi
MDS1(config)# iscsi enable module 1
MDS1(config)# iscsi enable
MDS1(config)# int gig1/2
MDS1(config-if)# no shut
MDS1(config-if)# int gi1/2.72
MDS1(config-if)# ip add 198.18.72.9 255.255.255.0
MDS1(config-if)# no shut
MDS1(config-if)# vrrp 72
MDS1(config-if-vrrp)# address 198.18.72.1
MDS1(config-if-vrrp)# prio 110
MDS1(config-if-vrrp)# no shut
MDS1(config-if-vrrp)# int iscsi1/2
MDS1(config-if)# no shut
MDS1(config-vsan-db)# vsan 10 interface iscsi1/2
Traffic on iscsi1/2 may be impacted. Do you want to continue? (y/n) [n] y
MDS1(config-vsan-db)#
We first enable the feature and configure the VRRP group. We make MDS1 the primary switch.
Which switch is the primary, is not given in this case in the question, so we can take any switch
we want for this use. Do not forget to put the iSCSI interface into another VSAN, because
VSAN 1 is still suspended and interface in that VSAN will not come online unless they are
changed to another VSAN which is unsuspended.
The question about the configuration that can only be done on MDS2, is about enabling the
distribution of the iSLB feature using the CFS protocol. This should be enabled.
MDS1
MDS1(config)# islb distribute
The next step in our task is that we need to enable the load balancing feature on the iSLB
activated switches. The load balancing feature only works in conjunction with the VRRP. It is
based on a ‘weight’ that is given to an initiator. The load balancing is always using the
number of initiators to load balance, not to the number of sessions on the switch.
You can specify a weight to an initiator to take care of the load balancing. When a host has
a lower bandwidth connection, it can be assigned a lower weight, as the amount of load it
generates to the MDS switch is lower than other servers. So the MDS can take more of these
smaller bandwidth servers on one interface.
Another use case for using load balancing is when there are certain initiators that you know are
going to set-up a lot of iSCSI sessions. Therefore more load is generated on the MDS and the
weight number should take care of that.
The default value is 1000.
Load balancing is enabled with only a single command.
MDS1
MDS1(config)# islb ?
abort Abort pending iSLB configuration
commit Commit pending iSLB configuration
distribute Enable CFS for iSLB configuration
initiator Configure iSLB initiator
save-initiator Make WWNs for iSLB initiator persistent
virtual-target Configure iSLB virtual-target
vrrp Configure iSLB on this vrrp group
zoneset Configure iSCSI zoneset activate
The load balancing can be monitored using the following output. All load balancing
configuration is distributed using the CFS Protocol, so both switches are aware of all changes
going on in the load balancing mechanism. This ensure that when the master fails, which keeps
track of the iSCSI login redirects, the backup switch will have all information regarding
the load balancing and is able to successfully take over the working of the switches. Another
redirect is sent to the iSCSI initiators and sessions are automatically re-initiated.
MDS1(config)# sh islb vrrp summ
--------------------------------------------------------------------------------
72 IPv4 Enabled
Interface GigabitEthernet1/2.72
Switch wwn: 20:00:00:0d:ec:54:a9:80
VRRP group id: 72, VRRP IP address: 198.18.72.1
Interface VRRP state: master
Interface load: 0
Interface redirection: enabled
Group redirection: enabled
Number of physical IP address: 1
(1) 198.18.72.9
Port vsan: 10
Forwarding mode: store-and-forward
Proxy initiator mode: disabled
iSCSI authentication: CHAP or None
Interface GigabitEthernet1/2.72
Switch wwn: 20:00:00:0d:ec:58:b1:aa
VRRP group id: 72, VRRP IP address: 198.18.72.1
Interface VRRP state: master
Interface load: 0
Interface redirection: enabled
Group redirection: enabled
Number of physical IP address: 1
(1) 198.18.72.11
Port vsan: 10
Forwarding mode: store-and-forward
Proxy initiator mode: disabled
iSCSI authentication: CHAP or None
MDS1(config)#
The master switch (VRRP master) will always redirect the first request towards the backup
switch. This is because the master switch always has more load, because it needs to handle all
iSCSI login requests. Therefore it automatically gives itself an invisible weight of 1000. This
means that when 2 requests with a weight of 1000 come in to the switch it will redirect both
to the backup switch. As the master is always less busy than the backup switches. This means
that in case the load of the switches is equal, the request is always redirected to the backup.
The next task is only a key thing to remember that with the following questions there are no
manual zoning changes allowed. This means that we should be using the auto-zoning
feature instead. This also means that we should be using ‘initiator-targets’, as this is the
only place where the auto-zoning will work. When you only configure an initiator and a
virtual-target, these will not be auto-zoned together, because the switch does not
know they belong to each other. Therefore we will be using the initiator target feature.
We will first configure as the next question states, the 5 initiators. Remember that we are
only allowed to configure MDS2 for initiators and targets.
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-1
MDS2(config-islb-init)# ?
metric Assign load balancing metric for the initiator
mutual-chap Assign username for initiator's challenge (mutual chap)
no Negate a command or set its defaults
static Create static nWWN or pWWN for the initiator
target Configure iSLB initiator target
username Assign username for iSLB login authentication
vsan Assign VSAN membership for the initiator
zonename Assign zone name for the initiator targets
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
We need to allocate system assigned nWWNs and 2 pWWNs to the initiators. Just like we are
able when configuring iSCSI initiators.
MDS2
MDS2(config-islb-init)# zonename ?
WORD Enter zone name (Max Size 55)
As the next question dictates we need to use zone names. By default the switch will
generate either a long hash or a name with ‘iSLB’ in it. You are able to make it more readable,
by configuring the zone name under the initiator. You are not allowed to use any dashes in
the zone names. Only underscores are allowed as special character.
After the commit has been initiated, the initiator is shown in the configuration. There you
can see that the MDS allocated a nWWN and 2 pWWNs to this initiator.
MDS2(config)# sh run | section "islb initiator"
MDS2(config)#
The next question in our task is that Host 3 is a database server with a lot of iSCSI
connections. As we discussed earlier. The load balancing that iSLB does is based on the
number of initiators that make a connection with the switch. This particular initiator
will have a lot of iSCSI sessions with the switch, meaning that it will generate a lot more load
on the switch compared to other initiators.
We do not know how many sessions are set-up compared to other initiators, but we just
need to ensure that the load balancing takes care of this heavier initiator. Therefore we will
be configuring a little higher load.
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-3
MDS2(config-islb-init)# metric 2000
MDS2(config-islb-init)# islb commit
Now we can verify our initiator configuration and see that everything configured has been
properly pushed to all switches in the fabric.
MDS1(config)# show islb initiator config
iSCSI Node name is iqn.islb-initiator-host-1
MDS1(config)#
The next 2 questions state that we need to configure the first iSLB virtual-target,
however we are not allowed to use the command required for this. Remember that you also
can not use any zoning changes to allow the traffic of the iSCSI initiator to the FC
target.
In this question we will be using the so-called ‘initiator targets’. These are targets that
are directly configured under the iSLB initiator and automatically only this initiator
can access the target. This eases the configuration of your initiators and it adds another
option for flexibility when you are configuring some disk for just one initiator. It will
automatically arrange the zoning for you by using the ‘auto-zoning’ feature.
In this case it’s not really efficient to use the initiator-target feature, because we have a
single target that is reached by multiple initiators.
Read ahead as you need to make this target high available with another target and use LUN
masking!
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-1
MDS2(config-islb-init)# ?
metric Assign load balancing metric for the initiator
mutual-chap Assign username for initiator's challenge (mutual chap)
no Negate a command or set its defaults
static Create static nWWN or pWWN for the initiator
target Configure iSLB initiator target
username Assign username for iSLB login authentication
vsan Assign VSAN membership for the initiator
zonename Assign zone name for the initiator targets
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
We need to configure this target in VSAN 10. Do not forget to add the initiator to VSAN
10 as well. Either by configuring the iSCSI interface inside this VSAN, which we already did, or
by configuring the VSAN command under the initiator.
When you do not do this, the initiator will not be active in this zone.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 ?
Now we configured the required LUN masking. The FC LUN 0x0 should be seen as 0xA.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb
You can configure the same IQN name, when you only change the LUN masking configuration.
This way your target is visible with 2 LUNs. This will also be visible in the show outputs.
MDS2
MDS2(config-islb-init)# islb zoneset activate
MDS2(config)# islb commit
The final command required is to make the auto-zoning active. This is not strictly necessary
as with CFS distribution enabled, the auto-zoning is automatically activated as soon as
the commit command is entered.
The commands for the initiator-targets are very lengthy, which might cause issues with
your terminal emulator. When you are running into issues, you can enable a longer
terminal width.
MDS2(config)# sh run | section islb
islb distribute
islb initiator name iqn.islb-initiator-host-1
static nWWN 21:01:00:05:9b:70:5a:42
static pWWN 21:02:00:05:9b:70:5a:42
Although the device-alias mode is enhanced. They are not kept in the configuration, but
are translated to the pWWN of the device-alias.
MDS2(config)# show islb initiator config
iSCSI Node name is iqn.islb-initiator-host-1
This verification command shows you in summary how the initiator target is automatically
configuring zones and how the 2 LUNs are visible and summarized in this single target.
Now we can configure the remaining initiators for the same target.
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb
vsan 10 fc-lun 0x1 iscsi-Configuring iSLB initiator target with device-alias J2D2
(VSAN 10) and sec-device-alias J1D2 (VSAN 10) fails: Initiator target name invalid
or duplicate (0x403c0044)
Now although this target is only available for the initiator where it is configured. You still
can not have overlapping symbolic-names in the configuration! Therefore we will add the
host number to the symbolic-name.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb2
MDS2(config-islb-init)# vsan 10
MDS2(config-islb-init)# exit
MDS2(config)# islb initiator name iqn.islb-initiator-host-3
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb3
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb3
MDS2(config-islb-init)# vsan 10
MDS2(config-islb-init)# exit
MDS2(config)# islb initiator name iqn.islb-initiator-host-4
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb4
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb4
MDS2(config-islb-init)# vsan 10
MDS2(config-islb-init)# exit
MDS2(config)# islb initiator name iqn.islb-initiator-host-5
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb5
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
sec-device-alias J1D2 sec-lun 0x1 sec-vsan 10 iqn-name iqn.jbod-disk2-islb5
MDS2(config-islb-init)# vsan 10
MDS2(config-islb-init)# islb commit
MDS2(config)#
Now we have configured our 2 LUNs to be highly available using auto-zoning in all
configured initiators!
The next 2 questions indicate another target that now should only be created under
iqn.islb-initiator-host-3. We don’t need to be worried about the high availability as it
will be by default!
However we can not use auto-zoning!
MDS2
MDS2(config-vsan-db)# islb initiator name iqn.islb-initiator-host-3
MDS2(config-islb-init)#
MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islb-
jbod1-disk1
Configuring iSLB initiator target with device-alias J1D1 (VSAN 20) fails: Initiator
target's primary pWWN is not in the same VSAN as the initiator (0x403c0060)
When you are configuring the initiator target, you must first configure the VSAN where
the initiator resides, before it will accept the command. You need to make the initiator
member of all VSANs that it needs to access targets in.
MDS2
MDS2(config-islb-init)# vsan 20
MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islb-
jbod1-disk1
MDS2(config-islb-init)# islb commit
Now we are not finished as we still need to create the zoning for this new initiator
target.
Here we can not use the IP address or the Symbolic Name of the initiator in the zone.
This means we need to use the statically assigned pWWN values or the nWWN. In this case we will
use the pWWN that is assigned by the MDS switch.
MDS2
MDS2(config)# zoneset name ZS1 v 20
Enhanced zone session has been created. Please 'commit' the changes when done.
MDS2(config-zoneset)# zone name ISLB_1
When you are using the pWWN for the zoning, keep in mind to add both of the pWWNs as the
initiator will get one out of 2 assigned, so you have to take care that either one can be the
active pWWN at that moment.
Now the final question of this task and this chapter is to configure authentication for all
initiators. This means that we need to configure the CHAP authentication on the
initiator.
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-1
MDS2(config-islb-init)# username host-1
MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-2
MDS2(config-islb-init)# username host-2
MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-3
MDS2(config-islb-init)# username host-3
MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-4
MDS2(config-islb-init)# username host-4
MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-5
MDS2(config-islb-init)# username host-5
MDS2(config-islb-init)# exit
MDS2(config)# username host-1 password iSLBpassw0rd iscsi
MDS2(config)# username host-2 password iSLBpassw0rd iscsi
MDS2(config)# username host-3 password iSLBpassw0rd iscsi
MDS2(config)# username host-4 password iSLBpassw0rd iscsi
MDS2(config)# username host-5 password iSLBpassw0rd iscsi
To finish we create the local authentication database entries, just like was required for the
iSCSI configuration.
Here we finished the chapter about Storage Networking Extensions over IP!
The next Chapter will do a deep dive into the subject of Unified Fabric. Where we will
focus on Fibre Channel technologies over Ethernet. Which are very different than the FCIP
technologies that we used in this chapter.
Take a break and be fresh to start the next chapter!
Chapter 7: Data
Center Unified
Fabric
Chapter 7: Data Unified Fabric is intended to let you be familiar with the Storage Networking features
available on the Cisco Nexus switches and combined with the Cisco MDS switches.
This chapter will be about implementing FCoE features inside of the Nexus switches and the backwards
compatibility with Native FC connections. Besides that we will be looking at N-Port Virtualization
configurations..
We highly recommend creating your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab. Multiple topology
drawings are available for this chapter.
General Rules
• Try to diagram out the task. Draw your own connections the way you like it
• Take a very close read of the tasks to ensure you don’t miss any points during grading!
• Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Solutions
The first questions in this initial task are to create some initial configuration. We need to make
sure that we leave configurations in tact on MDS1 and MDS2, because we will be re-using the
topology that we build in previous chapters.
The topology we will be using throughout this chapter is as the following drawing illustrates.
We will not be using VDC’s in this topology, but only the native switching. Except for some
special feature which we will be talking about in the next couple tasks.
The topology is using the 3 Nexus switches, the 2 MDS switches and the 2 JBOD arrays so we
can create a Fibre Channel fabric in combination with the Ethernet network of our lab.
This combination is the Cisco best practice for Datacenter networking, where the Distribution
switches (Nexus 5500) are connecting to both the Core switch (Nexus 7000) and Fibre Channel
Fabric (MDS switches).
We will be re-using the MDS configurations except that we need to disable the
GigabitEthernet interfaces as we do not need the FCIP connections anymore, which might
interfere with the next questions.
MDS1
MDS1(config)# interface gig1/1
MDS1(config-if)# shutdown
MDS1(config)# interface gig1/2
MDS1(config-if)# shutdown
MDS2
MDS2(config)# interface gig1/1
MDS2(config-if)# shutdown
MDS2(config)# interface gig1/2
MDS2(config-if)# shutdown
Now we need to extend our Fibre Channel fabric with native interfaces towards the Nexus 5500
switches.
The Nexus 5500 is the only switch in the Nexus series that supports native Fibre Channel
interfaces, besides the UCS 61xx fabric interconnects (which are not fully Nexus switches). The
first generation of Nexus 5000 switch (Nexus 5010 and Nexus 5020) supported FC uplinks
using dedicated uplink modules with either 10Gbps Ethernet (or FCoE) ports or dedicated
8Gbps Fibre Channel interfaces. These Fibre Channel interfaces are auto-switching for 8Gbps,
4Gbps, 2 Gbps and 1Gbps FC traffic, depending on the module (some only support up to
4Gbps).
With the second generation of Nexus 5000 series. Cisco introduced the concept of ‘universal
ports’. These universal ports are by default enabled for Ethernet and FCoE traffic, but
can be converted with a single command to native Fibre Channel interfaces. Of course you do
need to change the SFP in the port, before it will actually work. Fibre Channel has different
SFP’s than Ethernet SFP’s.
The interfaces are not licensed, but before you are able to use the Fibre Channel ports on the
Nexus 5500 chassis, you need to have a Storage Protocols Services License. This
license is valid for enabling both native Fibre Channel interfaces and FCoE on any of the ports in
the chassis.
You need to enable the Fibre Channel ports in the switch and this is a disruptive change! Every
additional Fibre Channel that you configure requires a reboot of the whole switch. This is
because the ASIC that is controlling the ports needs to be re-initialized and reprogrammed for
Fibre Channel traffic on that port.
There are limitations to which ports can be enabled for Fibre Channel traffic and which are not.
It means that you need to configure Fibre Channel ports next to each other. It’s not possible to
configure one interface as Ethernet and the next interface as Fibre Channel.
The following drawing illustrates an example in which some of the ports are enabled for Fibre
Channel and other ports are enabled for Ethernet, which is how you might want to cluster
certain ports that are serving a certain system.
The combination of FC and Ethernet is not possible to configure.
What is possible with the Unified Port technology is that the interfaces which can be
configured for Fibre Channel are on the right side of the switch. Meaning that all first ports can
be configured for Ethernet and the last ports for Fibre Channel.
This can not be mixed or turned around. There is no technical explanation for it, just was a
design decision of Cisco to do it this way.
The port layout on the switch will be like the following drawing illustrates.
You are able to configure from a single port to all ports as Fibre Channel ports, there is no
limitation to the amount of ports, just that they need to be next to each other.
Now we need to configure the Unified Ports on our switches.
Do not forget to do the reboot of the switch to make them active.
SW2
SW2(config)# slot 1
SW2(config-slot)# port 31-32 type fc
SW2(config-slot)# wry
[########################################] 100%
SW2(config-slot)# reload
SW3
SW3(config)# slot 1
SW3(config-slot)# port 31-32 type fc
SW3(config-slot)# wr
[########################################] 100%
SW3(config-slot)# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
After the reboot and you issue a show interface brief, you will see 2 new fc1/31 and
fc1/32 in your configuration to be configured. The SFPs in the ports are already of the Fibre
Channel type. So they will be immediately recognized and you are able to configure the native
Fibre Channel connection to the MDS switches.
Now when we read ahead in the number of questions we see that the interfaces should be
combined into a single connection as the FSPF protocol is only allowed to see a single
connection in its database. This means that we need to configure a port-channel.
Just like it’s possible to create ethernet port-channels, it’s also possible to configure Fibre
Channel port-channels just like on the MDS switches.
We first create our configuration for the MDS switches before the Nexus switches.
The configuration on the MDS switches is just equal to any other port, as the Nexus just acts as
a standard Fibre Channel Director/Switch like all other MDS switches. NX-OS is also a
rebranding of SAN-OS, so all FC code is still included in the OS and is not different in CLI than
the MDS switches.
MDS1
MDS1(config)# interface fc1/13
MDS1(config-if)# switchport mode e
MDS1(config-if)# channel-group 10
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/14
MDS1(config-if)# switchport mode e
MDS1(config-if)# channel-group 10
MDS1(config-if)# no shutdown
MDS1(config-if)# interface port-channel10
MDS1(config-if)# switchport mode e
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# switchport trunk allowed vsan 10
MDS1(config-if)# switchport trunk allowed vsan add 20
MDS1(config-if)# no shutdown
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface port-channel10
MDS2
MDS2(config)# interface fc1/13
MDS2(config-if)# channel-group 10
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/14
MDS2(config-if)# channel-group 10
MDS2(config-if)# no shutdown
MDS2(config-if)# interface port-channel10
MDS2(config-if)# switchport mode e
MDS2(config-if)# switchport trunk mode on
MDS2(config-if)# switchport trunk allowed vsan 10
MDS1(config-if)# switchport trunk allowed vsan add 20
MDS2(config-if)# no shutdown
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface port-channel10
Do not forget to enable the port-channel interface in another VSAN, because we are still using
the configuration from the other chapters the Default VSAN is still suspended and not able to
enable ports in.
Then we configure the Nexus switches so the port-channels will come online.
SW2
SW2(config)# int fc1/31
SW2(config-if)# channel-group 10
command failed: port not compatible [port mode]
Interfaces that are set to their default auto-mode are not allowed to be configured in a port-
channel. This means that we first need to manually set the port-mode before we can enable the
port in a port-channel
SW2(config-if)# sw mode e
SW2(config-if)# channel-group 10
fc1/31 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel,
then do "no shutdown" at both ends to bring it up
SW2(config-if)# no shut
SW2(config-if)# int fc1/32
SW2(config-if)# sw mode e
SW2(config-if)# channel-group 10
fc1/32 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel,
then do "no shutdown" at both ends to bring it up
SW2(config-if)# no shut
SW2(config-if)# int port-channel10
Channel group 10 is already a SAN port channel
^
% Invalid command at '^' marker.
The port-channel configuration on the Nexus 5000 series switches is not equal to the MDS
configuration for Ethernet and FC port-channels. You must use unique numbering for all port-
channels (both Ethernet and FC), but you cannot configure them with the same configuration.
For FC port-channels you need to use the special command ‘san-port-channel’ instead of
the regular port-channel command.
There you have the configuration available that you would expect to have with Fibre Channel
port-channels.
SW2(config-if)# int san-port-channel 10
SW2(config-if)# sw mode e
SW2(config-if)# sw trunk mode on
SW2(config-if)# sw trunk allowed vsan 10
SW2(config-if)# sw trunk allowed vsan add 20
SW2(config-if)# no shut
SW2(config-if)# vsan database
SW2(config-vsan-db)# vsan 10
SW2(config-vsan-db)# vsan 20
SW2(config-vsan-db)# vsan 10 interface san-port-channel 10
SW3
SW3(config)# int fc1/31
SW3(config-if)# sw mode e
SW3(config-if)# channel-group 10
fc1/31 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel,
then do "no shutdown" at both ends to bring it up
SW3(config-if)# no shut
SW3(config-if)# int fc1/32
SW3(config-if)# sw mode e
SW3(config-if)# channel-group 10
fc1/32 added to port-channel 10 and disabled
please do the same operation on the switch at the other end of the port-channel,
then do "no shutdown" at both ends to bring it up
SW3(config-if)# no shut
SW3(config-if)# int san-port-channel 10
SW3(config-if)# sw mode e
SW3(config-if)# sw trunk mode on
SW3(config-if)# sw trunk allowed vsan 10
SW3(config-if)# sw trunk allowed vsan add 20
SW3(config-if)# no shut
SW3(config-if)# vsan database
SW3(config-vsan-db)# vsan 10
SW3(config-vsan-db)# vsan 20
SW3(config-vsan-db)# vsan 10 interface san-port-channel 10
Now we configured both Nexus switches inside their own SAN Port-channel and gave them
membership to another VSAN, just to keep the configuration equal to the MDS switches to
prevent failures in the traffic when interfaces come online.
First we check if the interfaces are really seen in the Nexus chassis. This is an output of a pre-
configured Nexus 5000 switch.
SW2(config-if)# do sh int brief
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc2/1 4094 E on sfpAbsent -- -- 10
fc2/2 4094 E on sfpAbsent -- -- 10
fc2/3 4094 F on sfpAbsent -- -- --
fc2/4 4094 F on sfpAbsent -- -- --
fc2/5 4094 F on sfpAbsent -- -- --
fc2/6 4094 F on sfpAbsent -- -- --
fc2/7 4094 F on sfpAbsent -- -- --
fc2/8 4094 F on sfpAbsent -- -- --
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth1/1 1 eth trunk up none 1000(D) 3
Eth1/2 1 eth access down SFP not inserted 10G(D) --
Eth1/3 1 eth access down SFP not inserted 10G(D) --
Eth1/4 1 eth access down SFP not inserted 10G(D) --
Eth1/5 1 eth access up none 10G(D) --
Eth1/6 1 eth access up none 10G(D) --
Eth1/7 1 eth trunk up none 10G(D) --
Eth1/8 1 eth access down SFP not inserted 10G(D) --
Eth1/9 1 eth trunk up none 10G(D) --
Eth1/10 1 eth trunk up none 10G(D) --
Eth1/11 1 eth access down SFP not inserted 10G(D) --
Eth1/12 1 eth access down SFP not inserted 10G(D) --
Eth1/13 1 eth access down SFP not inserted 10G(D) --
Eth1/14 133 eth access up none 10G(D) --
Eth1/15 10 eth access down SFP not inserted 10G(D) --
Eth1/16 1 eth access down SFP not inserted 10G(D) --
Eth1/17 1 eth trunk down SFP not inserted 10G(D) 17
Eth1/18 1 eth trunk down SFP not inserted 10G(D) 18
Eth1/19 1 eth trunk up none 10G(D) 4096
Eth1/20 1 eth access up none 10G(D) --
--------------------------------------------------------------------------------
Interface Vsan Admin Status Oper Oper IP
Trunk Mode Speed Address
Mode (Gbps)
--------------------------------------------------------------------------------
san-port-channel 10 4094 on noOperMembers -- --
--------------------------------------------------------------------------------
Port-channel VLAN Type Mode Status Reason Speed Protocol
Interface
--------------------------------------------------------------------------------
Po3 1 eth trunk up none a-1000(D) lacp
Po17 1 eth trunk down No operational members auto(D) none
Po18 1 eth trunk down No operational members 10G(D) lacp
Po202 1 eth trunk down No operational members auto(D) lacp
Po1234 1 eth trunk down No operational members 10G(I) none
Po4096 1 eth trunk up none a-10G(D) lacp
--------------------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
--------------------------------------------------------------------------------
mgmt0 -- up 10.160.45.14 1000 1500
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
vfc1 1 E on trunking -- TE 10 --
SW2(config-if)#
You can see that the port-channels for both Ethernet and FC are separated from each other and
can be configured separately. The FC interfaces, although they are connected to the same ASIC
as the Ethernet interfaces are shown different from the normal Ethernet interfaces.
Then we verify our VSAN database membership.
SW2(config-vsan-db)# show vsan member
vsan 1 interfaces:
vsan 10 interfaces:
fc1/31 fc1/32 san-port-channel 10
vsan 20 interfaces:
SW2(config-vsan-db)#
Here we see that both the port-channel and the individual interfaces have members in the
VSAN that we configured it in.
Now we have a successful native FC connection between the MDS and the Nexus switches!
The next question states that we should request the lowest possible Domain ID.
In the previous chapters we configured parameters for giving out FC Domain ID’s that
prevent the switches to use very low numbers. Therefore we will automatically get higher
numbers assigned by configuring the lower numbers.
SW2
SW2(config)# fcdomain domain ?
<0-239> Configure the domain id in decimal, 0 not allowed for static dom
<0x0-0xef> Configure the domain id in hexadecimal
Do not forget to restart the FC Domain when you are changing Domain IDs. Without a restart
the domain ID’s will not be effective!
SW2
SW2(config)# fcdomain restart disruptive vsan 10
SW2(config)# fcdomain restart disruptive vsan 20
SW3
SW3(config)# fcdomain domain 0 preferred vsan 10
SW3(config)# fcdomain domain 0 preferred vsan 20
SW3(config)# fcdomain restart disruptive vsan 10
SW3(config)# fcdomain restart disruptive vsan 20
Now the switches will have assigned a higher number FC Domain ID, but have configured the
lowest possible value, which is 0.
Now we should be able to see all devices that are located in VSAN 10 and VSAN 20. Meaning
we are able to see all the JBOD disks in both VSANs and we should have zoning applied as this is
also taken care of when fabrics ‘merge’.
The final task is that we should keep in mind that there are already security mechanisms in
place in the VSANs. Both VSAN 10 and VSAN 20 are using port-security. So we should add
the Nexus switches SWWNs to those databases before they are fully operational!
We don’t have any hosts connected to the Nexus switches so the changes to the port-
security database is not that impacting as we only need to add the switch WWNs to the
port-security database to add the switches to the fabric.
Let’s check the port-security database of VSAN 10 on both MDS1 and MDS2 that we
configured in previous chapters.
MDS1
MDS1(config)# do sh port-sec database
--------------------------------------------------------------------------------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------------
10 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10 20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
10 20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)*
10 20:00:00:a3:fe:00:98:32(pwwn) Any
[Total 8 entries]
MDS2
MDS2(config-port-security)# do sh port-sec database
--------------------------------------------------------------------------------
Here we see that VSAN 10 is not using a distributed port-security database, so we need
to add the Nexus switches to both of the databases on the switches.
We will do this from the MDS point of view, again because it’s not distributed.
Check the SWWN of the 2 switches:
SW2
SW2# show wwn sw
Switch WWN is 20:00:00:05:9b:73:1b:c0
SW3
SW3# show wwn sw
Switch WWN is 20:00:00:05:9b:a4:1f:aa
Now we add those entries to the connecting MDS switches. In our topology SW2 is connected
to MDS1 and SW3 is connected to MDS2.
MDS1
MDS1(config)# port-security data vsan 10
MDS1(config-port-security)# swwn 20:00:00:05:9b:73:1b:c0
MDS2
MDS2(config)# port-security data vsan 10
MDS2(config-port-security)# swwn 20:00:00:05:9b:a4:1f:aa
Now the SWWN of the switches has been added to the port-security database and you are
good to go and have a working native Fibre Channel fabric for VSAN 10.
Now we shouldn’t forget about VSAN 20 as well, as this VSAN is also allowed on the trunk to
the MDS switches.
The port-security of VSAN 20 is distributed, so the change only needs to be done from one
MDS switch. In the previous chapter we were only allowed to do configuration changes on
MDS1 so we will honor this again in this question.
MDS1
MDS1(config)# do sh port-security data v 20
--------------------------------------------------------------------------------
VSAN Logging-in Entity Logging-in Point (Interface)
--------------------------------------------------------------------------------
20 c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
20 c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
20 c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
20 c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
20 21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*
[Total 5 entries]
MDS1(config)#
After activating the database on the MDS1, we can see an activated database on both MDS
switches with all necessary information for activating the security in VSAN 20.
Now we have finished our first task in this chapter by configuring native FC interfaces on the
Nexus series switches.
There are 2 very different types of FCoE. The first is the standard ‘F-port’ virtualization.
Where you connect your Fibre Channel environment to the Converged Network Adapter
(CNA) of a server. The Nexus then acts as a distribution switch for the first hop which is
converged and the next hop, the traffic is split in both an Ethernet uplink and a Fibre Channel
uplink.
The second type is the E-port virtualization. This is also called ‘multi-hop’ FCoE as you are
no longer converging only the first hop, but also further hops in the network. You can connect
Nexus switches using virtual E-ports to extend the Fibre Channel fabric through your Ethernet
network. In fact, you do not need a MDS switch in some topologies as they can perform all
features necessary, of course depending on the port density of your native Fibre Channel
environment.
In this first task about FCoE we will be using the converged F-Ports.
The initial question is related to the fact that we need to configure a server which is connected
to interface Ethernet1/24. We want to use a port-channel from the server to the network
and we are using the vPC technology for this use.
First we need to ensure that the vPC technology works again across the Nexus switches. We
will use the easiest solution, but configuring peer-keepalives across the mgmt0 interface
and one of the interlinks of the Nexus switches as the peer-link.
We did not get any requirements in this question so we can use any values we want!
You might see more of these types of questions in the lab, where you are free to use values you
want for the required configuration for this question. It lets you think about all the changes that
are necessary to complete the question.
We will configure the switches for the vPC feature first.
SW2
SW2(config)# feature vpc
SW2(config)# vpc domain 100
SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3
As we are using the mgmt0 interfaces as a source for the keepalives we do not need to
configure the source of the keepalives as it’s using the mgmt0 and the management VRF by
default as a source for the keepalives.
SW3
SW3(config)# feature vpc
SW3(config)# vpc domain 100
SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2
Now we need to finish the vPC peer configuration by configuring the peer-link. Keep in mind
that the peer-link needs to be a port-channel. In this case we only add one interface to it, as
we only require some communication.
SW2
SW2(config)# interface ethernet1/7
SW2(config-if)# channel-group 1001
SW2(config-if)# interface port-channel1001
SW2(config-if)# vpc peer-link
SW3
SW3(config)# interface ethernet1/7
SW3(config-if)# channel-group 1001
SW3(config-if)# interface port-channel1001
SW3(config-if)# vpc peer-link
Let’s verify if the switches see each other and are able to form vPCs.
SW3(config-if)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
Now we finish the first questions by configuring the vPC connecting the host.
SW2
SW2(config-if)# int e1/24
SW2(config-if)# sw mode trunk
SW2(config-if)# no shut
SW2(config-if)# channel-group 1002
SW2(config-if)# int po1002
SW2(config-if)# sw mode trunk
SW2(config-if)# no shut
SW2(config-if)# vpc 1002
SW2(config-if)#
SW3
SW3(config-if)# int e1/24
SW3(config-if)# sw mode trunk
SW3(config-if)# no shut
SW3(config-if)# channel-group 1002
SW3(config-if)# int po1002
SW3(config-if)# sw mode trunk
SW3(config-if)# no shut
SW3(config-if)# vpc 1002
SW3(config-if)#
Our vPC is online and is able to process traffic just like any other vPC!
Now we are going to configure the FCoE specifics. One very important topic to always keep in
mind is that you are required to have a strict distinction between Fibre Channel fabrics!
The Fibre Channel world is arranged 2 fabrics that are in no place connected to each other. This
has to do with the auto-learning nature of the Fibre Channel Network that devices can
interfere with each other when something goes wrong.
The hosts in the Fibre Channel network are smart enough to detect failures and failover to the
other fabric when this is necessary. This means that even we do have a virtual Port-Channel to
the connecting server, we still are not going to share Fibre Channel configuration across this
link.
The FC traffic is always carried in a dedicated VLAN. You cannot carry multiple VSANs over a
single VLAN. There is always a distinct VSAN to VLAN mapping. This mapping of course
needs to be equal on all switches in the topology. In this case we will not be using the 2 VSANs
across the switches, but we will keep the configuration of the VSAN to the local switch.
SW2
SW2(config)# feature fcoe
SW2(config-vlan)# vlan 10
SW2(config-vlan)# fcoe vsan 10
SW2(config-vlan)# vlan 20
SW2(config-vlan)# fcoe vsan 20
SW3
SW3(config)# feature fcoe
SW3(config-vlan)# vlan 10
SW3(config-vlan)# fcoe vsan 10
SW3(config-vlan)# vlan 20
SW3(config-vlan)# fcoe vsan 20
Next we configure the vPC interface for the FCoE traffic. We do this by first creating a virtual
interface. This virtual Fibre Channel interface (VFC) will be handling all Fibre
Channel communication.
To ensure you are only using one leg of the vPC you need to bind the individual interface
instead of the port-channel interface to this virtual FC interface.
SW2
SW2(config-vlan)# interface vfc1001
SW2(config-if)# bind interface Ethernet1/24
SW2(config-if)# no shutdown
SW2(config-if)# vsan database
SW2(config-vsan-db)# vsan 10 interface vfc1001
SW3
SW3(config-vlan)# interface vfc1001
SW3(config-if)# bind interface Ethernet1/24
SW3(config-if)# no shutdown
SW3(config-if)# vsan database
SW3(config-vsan-db)# vsan 20 interface vfc1001
This configuration enables the connection between the physical Ethernet interfaces. The Virtual
FC interface and the VSAN in which it belongs. Now the switch knows that it needs to use which
VLAN to tag the traffic when sending it across the physical interface.
We have a clear distinction across the Fibre Channel fabric, but still the Ethernet traffic is using
a virtual Port-Channel and can still load balance across the 2 interfaces.
Next we need to ensure that the switches discard frames that are not from their own fabric as
we are creating different FC fabrics.
This is done using the FC-MAP feature of the Nexus. By default the Nexus switches use a value
of 0E.FC.00 as the Fabric Identifier. When a switch does not belong to a fabric (so the
FC-MAP value is different) the switch discards frames.
You are able to configure FC-MAP values in the range of 0E.FC.00 up to 0E.FC.FF.
We will configure a different FC-MAP value on SW2 to ensure that it will drop traffic coming
from SW3.
SW2
SW2(config)# fcoe fcmap ?
<0xefc00-0xefcff> Enter FCMAP
The CLI help can be misleading as the 0 in front of the FC-MAP value needs to be typed in, but
the context help might indicate that it’s not necessary.
Next we are told to configure SW2 to be the primary switch. While the Fibre Channel fabrics can
be used independently. It’s still possible to configure the CNA of the Server to connect to one
particular fabric over the other switches.
This is controlled using the FCF priority. FCF stands for Fibre Channel Forwarder,
meaning a Fibre Channel Switch. This FCF priority is by default set to 128 and the lower
the value the more preferred the switch becomes.
Therefore we will lower this priority on SW2 to a very low value to ensure that it’s chosen to
become the preferred FCF switch in the fabric.
We can verify the settings by using a single show command.
SW2(config)# show fcoe
Global FCF details
FCF-MAC is 00:05:9b:73:1b:c0
FC-MAP is 0e:fc:02
FCF Priority is 1
FKA Advertisement period for FCF is 8 seconds
SW2(config)#
Here we see that we configured the FC-MAP and the FCF Priority like it it supposed to be.
The next task would be easily solvable by only allowing the FCoE VLANs on the trunk, but we
are not allowed to do this. The final question in this task is related to a feature specific to the
DCBX protocol.
The DCBX protocol can send messages regarding the VLANs that do not have a FCoE VSAN
enabled on them.
You do this by basically shutting down the LAN part of the interface.
SW2
SW2(config)# interface port-channel1001
SW2(config-if)# shutdown lan
SW3
SW3(config)# interface port-channel1001
SW3(config-if)# shutdown lan
By configuring this single command you signal to the CNA that only a few VLANs are allowed on
this trunk, which are enabled for FCoE. The other VLANs are not allowed to transfer any
packets across the link and the link basically became a Fibre Channel interface, with Ethernet
encapsulation!
Next we are going to use the FCoE feature for the multi-hop purpose.
MDS2
MDS2(config-if)# interface fc1/1
MDS2(config-if)# shutdown
MDS2(config-if)# interface fc1/2
MDS2(config-if)# shutdown
MDS2(config-if)# interface fc1/3
MDS2(config-if)# shutdown
MDS2(config-if)# interface fc1/4
MDS2(config-if)# shutdown
Now we broke our Fibre Channel fabric. This task is all about fixing this topology. In this task we
will be extending VSAN 20 across the Nexus switches.
In this case we will be using the same FCoE technology as we used in the previous task except
that in this case we will not be configuring F-ports, but E-ports to extend the fabric across
the Nexus switches.
There is no real other stuff happening other than the previous task where we configured a vPC
as an F-Port.
We need to ensure that VSAN 20 keeps functioning in the way that is shown in Drawing 2.
This means that we cannot use the interlink between SW2 and SW3 for traffic and we need to
use SW1 as well.
The native FC connections connecting SW2, SW3 and the MDS switches already works and VSAN
20 is working across this connection.
This means that we already have a good connection with our current Fibre Channel Fabric.
Next we are configuring the FCoE virtual E-ports that will connect the Nexus switches to
each other.
On the Nexus 7000 the architecture of FCoE is a little different than on the Nexus 5000 series.
First of all the Nexus 7000 does not support any native FC interfaces, but only supports to be a
FCoE forwarder. The Nexus 7000 also doesn’t support to be a Fibre Channel Bridge.
Which means that it’s not capable of encapsulating FC frames into Ethernet packets to become
FCoE frames. This of course has to do that there are no native FC packets coming into the
Nexus 7000 switch.
One other important change on the Nexus 7000 platform is that FCoE features are not
configured in the normal configuration. The control-plane of the Fibre Channel traffic is
separated from the Ethernet traffic. This is done because of the possibility to create Virtual
Device Contexts (VDCs) on the Nexus 7000 switches.
Normally you require a license to use VDC’s, but when you have the FCoE license you can
enable the Storage VDC as well.
The Storage VDC or FCoE VDC works different from other VDCs. Normal VDCs require you
to allocate interfaces to it. The Storage VDC works based on the EtherType of packets.
FCoE packets have a different EtherType on which the interface knows where to direct traffic
to.
The drawing below illustrates how the Storage VDC works and separates traffic coming in over
a single link.
You can only create a single Storage VDC where all FCoE traffic is terminated. The use of
normal Ethernet VDC’s is still supported. In this lab we will be using the global configuration
and one Storage VDC as this is the minimal configuration that is required to support FCoE on
the Nexus 7000.
To start our FCoE E-port configuration we start on the Nexus 5000s switches as they are
pretty much equal to the previous task where we configured F-port FCoE.
Drawing 2 illustrates that there are 2 connections between the Nexus 7000 and the Nexus
5000s. As the task did not explicitly state this, we shouldn’t have to do this, as we could also
create 2 E-ports between the Nexus switches, but this configuration is more commonly seen.
SW2
SW2(config)# interface ethernet1/11-12
SW2(config-if)# channel-group 1002 mode active
SW2(config-if)# no shutdown
SW2(config-if)# interface port-channel1002
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 20
SW2(config-if)# no shutdown
SW2(config-if)# exit
SW2(config)# interface vfc1
SW2(config-if)# bind interface port-channel1002
SW2(config-if)# switchport mode e
SW2(config-if)# switchport trunk allowed vsan 20
SW2(config-if)# no shutdown
SW2(config-if)# vsan database
SW2(config-vsan-db)# vsan 20 interface vfc1
SW3
SW3(config)# interface ethernet1/11-12
SW3(config-if)# channel-group 1002 mode active
SW3(config-if)# no shutdown
SW3(config-if)# interface port-channel1002
SW3(config-if)# switchport mode trunk
SW3(config-if)# switchport trunk allowed vlan 20
SW3(config-if)# no shutdown
SW3(config-if)# exit
SW3(config)# interface vfc1
SW3(config-if)# bind interface port-channel1002
SW3(config-if)# switchport mode e
SW3(config-if)# switchport trunk allowed vsan 20
SW3(config-if)# no shutdown
SW3(config-if)# vsan database
SW3(config-vsan-db)# vsan 20 interface vfc1
After this configuration the port-channel will come only, but the VFC interfaces will remain non-
operational as the other side has not been configured yet.
Now we move on by configuring the Nexus 7000 switch. The configuration is a little different
than on the Nexus 5000.
SW1-1
SW1-1(config)# install feature-set fcoe
We first need to enable a complete ‘feature-set’. This set enables a number of features, one
of which is VDCs and all Fibre Channel related processes.
SW1-1
SW1-1(config)# vdc StorageVDC type storage
SW1-1(config-vdc)# system default switchport
SW1-1(config-vdc)# feature lldp
Next we configure the VDC that we want to use for Storage traffic. We need to enable LLDP in
this VDC, because this protocol is used in the FIP (DCBX) protocol to identify which features are
possible with the end host.
SW1-1
SW1-1(config-vdc)# allocate fcoe-vlan-range 20
Then we enable which VLANs that are enabled for FCoE that need to be carried through the
VDC. So instead of issuing the allocate interface command in this place, you are using the
allocate VLAN command to ensure these VLANs are connected to the VDC and FCoE traffic is
sent to this special VDC.
SW1-1
SW1-1(config-vdc)# allocate shared interface ethernet2/1-4
SW1-1(config-vdc)# allocate shared interface port-channel1-2
SW1-1(config-vdc)# exit
The final step of the VDC configuration is to add the interfaces that are used for sharing the
Ethernet and FC traffic. Additionally with the VLAN configuration you need to configure the
shared interfaces.
Your configuration options are getting limited when you issue this command and you can only
configure a plain trunk connection then on the Ethernet port and no other features.
SW1-1
SW1-1(config)# license fcoe module 2
The next step is to enable one of the available licenses in the chassis on the module. Without an
FCoE license enabled on the module the feature will not work. Keep in mind that the FCoE
feature only works on F-type linecards! In the CCIE Data Center lab we only use the first
generation of linecards so we have a M1 and a F1 linecard available in our chassis. In this lab we
are using the F-type line card. This is also the reason why we are using different interfaces
than we are used to on the Nexus 5000s as uplinks to the F1 linecard.
Then we configure our Ethernet interfaces with a trunk connection and the port-channel
configuration required to support the connectivity to the Nexus 5000s.
SW1-1
SW1-1(config)# vlan 20
SW1-1(config-vlan)# exit
SW1-1(config)# interface ethernet2/1-2
SW1-1(config-if)# channel-group 1 mode active
SW1-1(config-if)# no shutdown
SW1-1(config-if)# interface port-channel1
SW1-1(config-if)# switchport mode trunk
SW1-1(config-if)# switchport trunk allowed vlan 20
SW1-1(config-if)# no shutdown
SW1-1(config-if)# exit
SW1-1(config)# interface ethernet2/3-4
SW1-1(config-if)# channel-group 2 mode active
SW1-1(config-if)# no shutdown
SW1-1(config-if)# interface port-channel2
SW1-1(config-if)# switchport mode trunk
SW1-1(config-if)# switchport trunk allowed vlan 20
SW1-1(config-if)# no shutdown
SW1-1(config-if)# exit
Now we step into the VDC and create the configurations there related to the FCoE protocol.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
Within the Storage VDC we must enter the configuration prompt again.
SW1-1
SW1-1-StorageVDC(config)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20
SW1-1-StorageVDC(config-vsan-db)# exit
We open up all the interfaces that are shared with this VDC and then connect the VSAN 20 to
the VLAN that is also shared with this VDC. Just like we would on the Nexus 5000s.
SW1-1
SW1-1-StorageVDC(config)# interface vfc1
SW1-1-StorageVDC(config-if)# bind interface port-channel1
SW1-1-StorageVDC(config-if)# switchport mode e
SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20
SW1-1-StorageVDC(config-if)# no shutdown
SW1-1-StorageVDC(config-if)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc1
SW1-1-StorageVDC(config-vsan-db)# exit
We create the Virtual E-port on a VFC interface bound to the port-channel interface
connecting to SW2.
After this configuration and we apply the VSAN database configuration we have enabled our
fabric to being extended in to SW1-1 just like we want.
SW1-1
SW1-1-StorageVDC(config)# interface vfc2
SW1-1-StorageVDC(config-if)# bind interface port-channel2
SW1-1-StorageVDC(config-if)# switchport mode e
SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20
SW1-1-StorageVDC(config-if)# no shutdown
SW1-1-StorageVDC(config-if)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc2
When finishing the second interface connecting to SW3 we finished the lengty configuration.
The FCoE configuration is successfully applied and we are able to see the fabric being extended
across multiple Nexus switches.
On MDS1 we should see the JBOD connected to MDS2 and vice versa!
The next question states that we should protect against loopbacks. What is possible with VFC
connections is that interfaces are created that are looped back to the same switch.
To prevent this, you can enable a mechanism that checks for the VF ID in the FCoE packets
and when this matches to the switch its own VFID it will reject the traffic.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
SW1-1-StorageVDC(config)# fcoe veloopback
The next couple questions are all related to each other. We need to configure link
authentication, just like we did in the Fibre Channel Networking chapter. This
technology is called FC-SP, where a hashed authentication key is used to send a challenge to
the other switch to authenticate the interface.
This technology is also available on the Nexus switches when you are using FCoE interfaces.
An important topic is the combination of interface modes. You can configure an interface with
different modes as it might be required that you configure an interface with optional
authentication. The table below demonstrates the different modes and what will happen when
these 2 interfaces are connected to each other.
The On and Off modes are simple modes that do what they tell. They enforce authentication or
reject it. The auto modes are there for optional authentication. There should be at least 1
active link when both sides are on auto as the passive link will wait for an authentication
request before starting any negotiation.
Our question states that SW1-1 should never start the negotiations for authentication. The
other questions state various comments about which passwords are used.
Read these carefully as they all contain vital information to finish the task.
First look up the SWWNs of all the 3 switches so you have the information to create the
authentication entries.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# show wwn switch
Switch WWN is 20:00:00:05:ab:a3:16:76
SW2
SW2# show wwn sw
Switch WWN is 20:00:00:05:9b:73:1b:c0
SW3
SW3# show wwn sw
Now we start by configuring the passwords used to authenticate the switches towards each
other.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
SW1-1-StorageVDC(config)# feature fcsp
SW2
SW2(config)# feature fcsp
SW2(config)# fcsp dhchap password SecureNexus5000-1
SW3
SW3(config)# feature fcsp
SW3(config)# fcsp dhchap password IPexpertIsAwesome
Then we continue by configuring the switches to authenticate the devices using their specified
passwords.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
SW1-1-StorageVDC(config)# fcsp dhchap device 20:00:00:05:9b:73:1b:c0 password
SecureNexus5000-1
Same accounts for SW3 which is authenticated with the specified password.
To finish we configure the interfaces to support the FC-SP configuration without actively
starting to negotiate it.
SW1-1
SW1-1-StorageVDC(config)# interface vfc1
SW1-1-StorageVDC(config-if)# fcsp auto-passive
SW1-1-StorageVDC(config)# interface vfc2
SW1-1-StorageVDC(config-if)# fcsp auto-passive
Configuration of both switches is even equal as it’s using the same password for both switches.
SW2
SW2(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password
SW2(config)# interface vfc1
SW2(config-if)# fcsp auto-active
SW3
SW3(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password
SW3(config)# interface vfc1
SW3(config-if)# fcsp auto-active
Now our interfaces are successfully authenticated using a secure hash, but wait we should
enable a SHA-1 hash! By default the MD5 hash is enabled, so we don’t comply with the full
questioning.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
SW1-1-StorageVDC(config)# fcsp dhchap hash sha1
SW2
SW2(config)# fcsp dhchap hash sha1
SW3
SW3(config)# fcsp dhchap hash sha1
Now when all links are flapped again, they will be authenticated using a SHA-1 hash of the
correct key and we answered the questions correctly.
To finish this task we need to enable a mechanism so only the active switches are allowed in
the Fibre Channel Fabric of VSAN 20.
This means that we are going to use Fabric Binding. This was introduced in one of the
latest versions of NX-OS, but is now fully supported. Pay attention that Fabric Binding is only
supported in the SAN Enterprise license on the Nexus series and the Mainframe
license on the MDS switches!
Keep in mind that you need to configure this on all the switches as the fabric-binding feature
can not be distributed using the Cisco Fabric Services (CFS), which are also available
on the Nexus switches.
SW2
SW2(config)# feature fabric-binding
SW2(config)# fabric-binding data vsan 20
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
SW2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
SW2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
SW2(config-fabric-binding)# exit
SW2(config)# fabric-binding activate v 20
SW3
SW3(config)# feature fabric-binding
SW3(config)# fabric-binding data vsan 20
SW3(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
SW3(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
Don’t forget to include the MDS switches as well, as they are still part of the fabric too!
MDS1
MDS1(config)# feature fabric-binding
MDS1(config)# fabric-binding data vsan 20
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
MDS1(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
MDS1(config-fabric-binding)# exit
MDS1(config)# fabric-binding activate v 20
MDS2
MDS2(config)# feature fabric-binding
MDS2(config)# fabric-binding data vsan 20
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
MDS2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
MDS2(config-fabric-binding)# exit
MDS2(config)# fabric-binding activate v 20
Now we completed our task and have a successful multi-hop FCoE network running for
VSAN 20. The next task will be about optimizing this network for FCoE which has some strict
QoS requirements for the network!
By default this value is CoS 3. This can be changed, but this is not recommended.
Priority Flow Control can be enabled on a per link basis, but this is not necessary in most
scenarios as the DCBX protocol already negotiates this behavior with the connected host or
switch. This negotiation is accepted by the other end and the CoS value on which to apply PFC
is configured.
The first question we see is that we should make our topology to support FCoE on all Nexus
switches in which we should follow the best practices.
There are a number of options possible to configure on the Nexus 7000 switches. By default
when you enable the FCoE feature, it will not change anything about the QoS behavior of the
device. This needs to be manually changed. The Nexus 7000 has a couple of built-in QoS
behaviors that can be altered.
By default a template is active which is not honoring the No Drop behavior of CoS 3. Then
there are multiple templates available where you are able to configure an even deeper level of
Priority No Drop QoS, which might be necessary to place certain critical FCoE systems in,
so they are getting prioritized over other FCoE traffic. Or it can be used for Control traffic that
may not be dropped.
The table below demonstrates the CoS to Queue mapping on the Nexus 700 with the available
default templates.
Template Default QoS Priority QoS No Drop Qos Priority No Drop QoS
default-nq-8e-policy 0,1,2,3,4 5,6,7 - -
The first change we need to do is to configure the Nexus 7000 switch to actually support FCoE
queues in its configuration. By default the QoS policy on the Nexus 7000 does not incorporate a
FCoE queue. So we should enable this.
We will pick the default-nq-7e-policy for this use, as this is the most standard No Drop
configuration and we do not require any additional queues to be mapped as well.
SW1-1
SW1-1(config)# system qos
SW1-1(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
SW1-1(config-sys-qos)# exit
The next step is to enable the No Drop queue in the QoS configuration. The policy that is used
is one which is by default available on the Nexus 7000s. You need to enable a No Drop queue
for CoS 3 on the entire switch when enabling FCoE. Without this configuration the FCoE
feature will not work correctly and especially in terms of congestion, the FCoE traffic will not be
lossless!
One of the nice things of the Nexus 5000 series is that it automatically is enabled to support
FCoE. There are 2 QoS groups by default enabled for queueing and one is selected to have the
no-drop queue enabled.
SW2(config-if)# show queuing interface
Ethernet1/1 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 50
1 WRR 50
RX Queuing
qos-group 0
q-size: 243200, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port : 882892828
Ucast pkts sent to the cross-bar : 859238738
Mcast pkts sent to the cross-bar : 23654090
Ucast pkts received from the cross-bar : 213450997
Pkts sent to the port : 229752814
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 1
q-size: 76800, HW MTU: 2240 (2158 configured)
drop-type: no-drop, xon: 128, xoff: 240
Statistics:
Pkts received over the port : 0
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
match cos 3
The default policy-maps are configured to match on the class-maps and to apply appropriate
queueing and MTU sizes.
SW2(config-if)# show policy-map
Interfaces have these policies applied by default. Which makes it very easy for a QoS best
practice configuration on the Nexus 5000.
port-channel3
Now we know that QoS is enabled on the switches and that they all take care of FCoE traffic
with the No-drop queue we even included the last question of this task as well, because with
the adaption of the default QoS policies we also included the MTU statements.
Within the Nexus switches it’s not necessary to change the MTU size on the interfaces directly.
You can change MTU sizes per queue on the Nexus. This means that the FCoE queue can have a
different MTU than all the others.
This is automatically arranged by the templates that we apply as that the FCoE class will get the
MTU size to support maximum frame size of the Fibre Channel packet, which is 2112.
The next question about blocking receivers. The Nexus knows a special mechanism to use for
QoS and queuing. It uses a technology called Virtual Output Queuing. This technology
uses the ingress buffers of all interfaces to create a very large pool of Egress buffers.
It means that traffic is held at the ingress port, until the egress port has time to transmit the
packet. This is arranged by the ‘token’ principle that the Nexus switches use. Before traffic is
sent across the switch fabric between ASICs and/or linecards. The traffic needs to have a
token before it’s allowed to go over the fabric. Now the packet is stalled at the ingress port,
until it receives a token to be transmitted over to the egress interface.
This Virtual Output Queueing Unicast limit is required to enable the feature that we are
looking for in this question. Which is enabled with a simple single command.
SW2
SW2(config)# hardware unicast voq-limit
The last question of this task introduces another important topic about FCoE traffic. The native
Fibre Channel protocol knows the concept of B2B credits (Buffer to Buffer credits)
to ensure a lossless behavior of the link. When you are using a long link on a Fibre Channel
switch, it means you require more B2B credits than before, because there are more packets
in transit on an interface at any given time, so it takes more time before acknowledgements are
received for these packets and before you are able to send additional packets onto the link.
Within FCoE there is not a concept of B2B credits, but there is the concept of Pause frames
to hold up the sending of packets until further notice.
Now this Pause frame is sent by the receiver towards the sender. When you have a long link, it
means that at the time the receiver has sent this message, the sender already sent more
packets onto the link, because there are more packets in transit. This might cause a buffer
overflow and might cause the switch to have to drop packets.
To overcome this, we need to ensure the buffer size is set to the correct size according to the
link size. The Nexus 5000 and 7000 have strict limitations on the length of an FCoE link as there
is not much buffer space available on those platforms.
On the Nexus 5500 where we are configuring these values the maximum supported link-size is
up to 3000 meters or 3 kilometers. We need to configure the buffer size, the buffer fill
rate at which it should start sending Pause frames, because this needs to be done before the
buffer is full and finally the buffer rate at which it can send a frame so traffic may be sent again.
We need to perform this configuration on both SW2 and SW3. Within the documentation there
is an example configuration for 3000 meter links. When we divide this by 3 and multiply by 2,
we have the buffer sizes for 2000 meter links. Which are the values we are going to use.
When we divide these by 3 and multiply by 2 we get the values for 2000 meter links.
3000 meter values:
We can not change the default policy-map, so we need to create a new policy-map of which we
base our configuration from the current default.
Then we apply our special buffer sizes.
SW2
SW2(config)# policy-map type network-qos 2000m-nq-policy
SW2(config-pmap-nq)# class type network-qos class-fcoe
SW2(config-pmap-nq-c)# pause no-drop
SW2(config-pmap-nq-c)# mtu 2158
SW2(config-pmap-nq-c)# pause no-drop buffer-size 101333 ?
pause-threshold Buffer limit for pausing in bytes
port-channel3
RX Queuing
qos-group 0
q-size: 218560, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1366
Statistics:
Pkts received over the port : 883236801
Ucast pkts sent to the cross-bar : 859564057
Mcast pkts sent to the cross-bar : 23672744
Ucast pkts received from the cross-bar : 213458097
Pkts sent to the port : 229759946
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 1
q-size: 101440, HW MTU: 2240 (2158 configured)
drop-type: no-drop, xon: 690, xoff: 83
Statistics:
Pkts received over the port : 0
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
We see that the policy-map is correctly generated and now applied to all interfaces, because
we see different queue sizes and other xon/xoff values!
We successfully completed task 4!
By default when you configure an F-port where an N-port is connected to. The MDS switch
only allows up to 1 device to log-in to it. Except for Fabric Loop (FL) ports, where multiple
devices in the loop can log-in. Except that this last technology is very different.
We want normal F-ports to log-in multiple times to a single link on a smart MDS switch in the
core network.
This is where we start using “N-Port Virtualization” or NPV. This technology allows
devices to log-in to a fabric and then translates this request up to a core switch on a dedicated
uplink connection.
The core switch also requires some special configuration. The connections down to the
access switch (or edge switch) are just default F-ports, but a feature called N-port ID
Virtualization (NPIV) needs to be enabled to accept the translated log-ins from the
access switch in to the core switch.
The NPIV feature on the core switch enables the fact that it allows for handing out multiple
FCIDs to a single F-port.
This task and the next one will be using a little different topology to support the NPV features.
NPV requires a reboot of the device after which it can no longer be used as a normal MDS
switch. The chances of seeing this technology used on the CCIE Data Center lab is small,
because it would cost a whole MDS switch just for this single feature.
The drawing below is a subset of Drawing 3, which the topology we are using for this task.
The JBODs are the multiple hosts that want to connect to MDS1 where MDS2 should handle
their logins.
The terminology in NPV environments is that the switch where devices are connected is called
the “edge” switch and where the log-ins are handles is the “core switch”.
The argument in this task that the fabric should support more than 239 switches per VSAN
comes from the fact that you can not have more than 239 Domain ID’s handed out in your
Fibre Channel Fabric. This is due to the fact that this is a single byte number, with some
reservations.
The NPV edge switch does not consume a FC Domain ID as it’s not really participating in the
Fabric. Which introduces the possibility of having over 239 switches in the Fabric (VSAN).
We should first enable the ISL links again between the switches.
MDS1
MDS1(config-if)# interface fc1/1
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/2
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/3
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/4
MDS1(config-if)# no shutdown
MDS2
MDS2(config-if)# interface fc1/1
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/2
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/3
MDS2(config-if)# no shutdown
MDS2(config-if)# interface fc1/4
MDS2(config-if)# no shutdown
After that we enable the NPIV feature on the core switch, so it allows handing out multiple
FCIDs on a single interface.
MDS2
MDS2(config-if)# feature npiv
Next we should make MDS1 the edge switch. When you enable NPV it automatically does a
write erase of the whole switch and performs a reboot. This is a very impacting command!
MDS1
MDS1(config)# feature npv
Verify that boot variables are set and the changes are saved. Changing to npv mode
erases the current configuration and reboots the switch in npv mode. Do you want to
continue? (y/n):y
PowerPC
CPU: 8548E, Version: 2.1, (SVR:0x80390021)
Core: E500, Version: 2.2, (PVR:0x80210022)
Clocks: CPU:1333 MHz, CCB: 533 MHz,
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down
(Serial number QCS1427K1CM)
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number
QCS1427K1NK)
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Next we configure the interfaces to support the topology. We will be using all the interfaces
that we have without bundling them. Traffic is able to load balance across the available uplinks
towards the core switch.
MDS2
MDS2(config-if)# interface fc1/1
MDS2(config-if)# switchport mode f
MDS2(config-if)# interface fc1/2
MDS2(config-if)# switchport mode f
MDS2(config-if)# interface fc1/3
MDS2(config-if)# switchport mode f
MDS2(config-if)# interface fc1/4
MDS2(config-if)# switchport mode f
Default F ports are used for connecting core to edge switches in the NPV topology.
We then ensure that the interfaces are placed into VSAN 10 as that is what the task states.
MDS2
MDS2(config-if)# vsan database
MDS2(config-vsan-db)# vsan 10 interface fc1/1
MDS2(config-vsan-db)# vsan 10 interface fc1/2
MDS2(config-vsan-db)# vsan 10 interface fc1/3
MDS2(config-vsan-db)# vsan 10 interface fc1/4
Now that we configured our core switch to support our topology we focus on the NPV edge
switch MDS1 again.
We configure the uplinks to the core switch to be NPV uplinks.
MDS1
MDS1(config)# int fc1/1-4
MDS1(config-if)# sw mode np
MDS1(config-if)# no shut
MDS1(config-if)#
Once this is enabled the interfaces come only on the core switch and the NPV switch performs
an FLOGI into the core.
MDS2(config-vsan-db)# do sh int fc1/1
fc1/1 is up
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:05:00:05:9b:77:0d:c0
Admin port mode is F, trunk mode is on
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x1e0000 N 20:06:54:7f:ee:18:07:a0 (Cisco) npv
0x1e0020 N 20:05:54:7f:ee:18:07:a0 (Cisco) npv
0x1e0040 N 20:04:54:7f:ee:18:07:a0 (Cisco) npv
0x1e0060 N 20:03:54:7f:ee:18:07:a0 (Cisco) npv
Now the switch is ready to receive traffic and log-ins! Let’s verify on the NPV switch.
MDS1(config-if)# show npv status
npiv is disabled
External Interfaces:
====================
Interface: fc1/5, State: Failed(Mismatch in VSAN for this upstream port)
Interface: fc1/6, State: Failed(Mismatch in VSAN for this upstream port)
Server Interfaces:
==================
We see a mismatch in VSAN configuration. The NPV switch is still VSAN aware and you are even
able to configure multiple VSANs on the NPV switch when this would be necessary.
Therefore we also need to do our VSAN configuration on the NPV switch.
MDS1
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10
MDS1(config-vsan-db)# vsan 10 interface fc1/1
MDS1(config-vsan-db)# vsan 10 interface fc1/2
MDS1(config-vsan-db)# vsan 10 interface fc1/3
MDS1(config-vsan-db)# vsan 10 interface fc1/4
After configuring the VSAN database the status looks fine and the NPV switch is now really
ready to accept logins and traffic from end hosts.
MDS1(config-vsan-db)# show npv status
npiv is disabled
External Interfaces:
====================
Interface: fc1/1, VSAN: 10, FCID: 0x1e0020, State: Up
Interface: fc1/2, VSAN: 10, FCID: 0x1e0000, State: Up
Interface: fc1/3, VSAN: 10, FCID: 0x1e0040, State: Up
Interface: fc1/4, VSAN: 10, FCID: 0x1e0060, State: Up
Server Interfaces:
==================
MDS1(config-vsan-db)#
Now it’s time to configure our JBODs to be logging in to the NPV switch.
MDS1
MDS1(config-if)# interface fc1/5
MDS1(config-if)# switchport mode f
MDS1(config-if)# no shutdown
MDS1(config-if)# interface fc1/6
MDS1(config-if)# switchport mode f
MDS1(config-if)# no shutdown
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface fc1/5
MDS1(config-vsan-db)# vsan 10 interface fc1/6
When you would forget to add the interfaces to the VSAN database, the ports will not come
online and will give you this message.
MDS1(config)# show int fc1/5
fc1/5 is down (NPV upstream port not available)
Now when we verify our Core switch we see FLOGI’s being performed against the switch, just
like we would expect from any other end host system.
MDS1(config-vsan-db)# show flogi data
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc1/1 10 0x1e0020 20:05:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1
fc1/1 10 0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54
fc1/2 10 0x1e0000 20:06:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x1e0000 N 20:06:54:7f:ee:18:07:a0 (Cisco) npv
0x1e0020 N 20:05:54:7f:ee:18:07:a0 (Cisco) npv
0x1e0100 N 21:01:00:e0:8b:2a:f0:54 (Qlogic) scsi-fcp:init
Now the zoning and all other features related to this end host are applied to the core switch
and not to the edge switch. The edge switch really is nothing different than some sort of port
extender of the network.
On the NPV switch it’s also visible that the hosts are logging in to the switch.
MDS1(config-if)# sh npv status
npiv is disabled
External Interfaces:
====================
Interface: fc1/1, VSAN: 10, FCID: 0x1e0020, State: Up
Interface: fc1/2, VSAN: 10, FCID: 0x1e0000, State: Up
Interface: fc1/3, VSAN: 10, FCID: 0x1e0040, State: Up
Interface: fc1/4, VSAN: 10, FCID: 0x1e0060, State: Up
Server Interfaces:
==================
Interface: fc1/5, VSAN: 10, State: Up
Interface: fc1/6, VSAN: 10, State: Up
MDS1(config-if)#
The next question is related to steering traffic of the NPV switch across different uplinks.
By default traffic will be balanced across the uplinks, but only once when they come online.
There will be no second time the traffic is load balanced.
It’s possible to steer traffic manually across different uplinks of the NPV switch. This is done
using Traffic-Maps. There you are able to specify which server interface needs to talk with
which uplink interface.
You can first check the current load balancing configuration with the following outputs.
MDS1(config-if)# show npv traffic-map
----------------------------------------
fc1/5 fc1/1
fc1/6 fc1/1
----------------------------------------
MDS1(config-if)#
SW2
SW2(config)# feature fcoe-npv
Do not forget to enable NPIV on SW3 otherwise it will not accept multiple logins through a
single F-port.
SW3
SW3(config)# feature npiv
We will configure the fourth uplink to SW3 as the uplink and VSAN 20. In relation to the
previous tasks we will be using VLAN 20 for the VSAN to VLAN mapping.
SW2
SW2(config)# interface e1/8
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 20
SW2(config-if)# no shutdown
SW2(config-if)# vlan 20
SW2(config-vlan)# fcoe vsan 20
SW2(config-vlan)# exit
SW2(config)# interface vfc8
SW2(config-if)# bind interface e1/8
SW2(config-if)# switchport mode np
SW2(config-if)# no shutdown
SW3
SW3(config)# interface e1/8
SW3(config-if)# switchport mode trunk
SW3(config-if)# switchport trunk allowed vlan 20
SW3(config-if)# no shutdown
SW3(config-if)# vlan 20
SW3(config-vlan)# fcoe vsan 20
SW3(config-vlan)# exit
SW3(config)# interface vfc8
SW3(config-if)# bind interface e1/8
SW3(config-if)# switchport mode f
SW3(config-if)# no shutdown
Chapter 8: Security
Features
Chapter 8: Security Features is intended to let you be familiar with the Security features which are
available on the Nexus platform. You will be configuring both AAA services and other management
security as well as LAN security features like DHCP snooping and other protective features.
We highly recommend creating your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab. Multiple topology
drawings are available for this chapter.
General Rules
Try to diagram out the task. Draw your own connections the way you like it
Take a very close read of the tasks to ensure you don’t miss any points during grading!
Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular
chapter
Solutions
One important topic is Cisco TrustSec. This technology has been introduced on the Nexus
series switches to introduce security within the datacenter.
This technology is only available on a selected number of devices and ensures you can create a
multi-tenant environment. Traffic is identified with tags. All devices that are using this tag are
authenticated between each other and traffic that is sent between the devices is secured using
encryption, anti-replay protection and message integrity checks.
The second major task of this chapter is the Access Control Lists (ACL’s) where you
can protect VLANs, Interfaces and Layer 3 VLAN interfaces. With ACLs you are able to protect
an entity according to the Source and/or Destination IP address, IP protocol and
Source and/or Destination TCP/UDP port number. VLANs can be protected by filtering
on MAC addresses (source and/or destination).
This initial task will be about configuring a feature which has been available on Catalyst
switches for a long time and is now introduced on the Datacenter switches as well.
The port security feature allows you to configure a maximum number of hosts that are
allowed to connect to the interface. Hosts mean in this case MAC addresses. These MAC
addresses can be auto learnt or statically configured. In a data center environment you can
easily have thousands of MAC addresses so this is not often seen in these environments.
One flexibility in the feature that has been introduced in the Nexus switches is that it’s possible
to configure the port-security maximum amounts per VLAN. So you are able to configure
port-security on trunk links as well and specify maximums per VLAN.
There are a lot of options with this feature so therefore there are a high number of questions in
this task.
There are limitations to this feature. Some are system limits and others are configurable.
8192 is the maximum amount of MAC addresses that can be secured by the port security
feature on an entire system
1024 is the maximum amount of MAC addresses that can be learnt on any interface
You can configure a maximum per VLAN, but this is not configured by default
The feature not only protects against too much hosts logging in to the same interface, it also
protects against the moving of MAC addresses.
This means that when a certain host is learnt or configured on an interface and it is connected
to another interface. This is seen as a violation and the host is denied access or the interface
is shutdown. This is called a ‘MAC Move Violation’.
The feature does not work for vPC interfaces as the database can not be synchronized
between vPC peer switches. This introduces violations when hosts move between member
interfaces of the vPC, therefore this command is not available on vPC interfaces. It is available
on standard port-channels, where it doesn’t matter to which member interface the host is
accessed.
The first questions in this task are related to setting up the Nexus switches for configuration.
We need to configure the management network and some hostnames, because the switches
boot up with a blank configuration. This is something we did a number of times, so it shouldn’t
be a surprise.
SW1
This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Nexus Switch
10.2.8.74 login: admin
Password: IPexpert123
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
switch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# switchname SW1
SW1(config)# interface mgmt0
SW1(config-if)# ip address 172.16.100.1/24
SW1(config-if)# no shutdown
SW1(config-if)# ip default-gateway 172.16.100.254
SW1(config)# ntp server 172.16.100.254
After configuring the management interface and the hostname, we are good to go. The other 2
switches are configured in a similar way.
SW2
Enter the password for "admin": IPexpert123
Confirm the password for "admin": IPexpert123
This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Nexus Switch
10.2.8.75 login: admin
Password: IPexpert123
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
switch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# switchname SW2
SW2(config)# interface mgmt0
SW2(config-if)# ip address 172.16.100.2/24
SW2(config-if)# no shutdown
SW2(config-if)# ip default-gateway 172.16.100.254
SW2(config)# ntp server 172.16.100.254
SW3
Enter the password for "admin": IPexpert123
Confirm the password for "admin": IPexpert123
This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Nexus Switch
10.2.8.76 login: admin
Password: IPexpert123
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
Now we start with configuring the logical topology of this lab. This logical topology is fairly
simple as we do not require much topology to test the security features.
In this lab we will simulate a lot of hosts behind interfaces, but they are not really connected.
You will see similar questions in the CCIE Data Center lab exam.
The topology is created with standard LACP port-channels between all switches and a
number of client interfaces where simulated hosts are then ‘connected’.
Below is Drawing 2 which is used as the logical topology in this chapter.
Within this chapter we will be configuring several VLANs. When VLAN numbers are mentioned
and do not exist yet you should create them on the fly while working through the task.
We will start by configuring the port-channels between the switches. Ensure you are using
LACP as this is a requirement in the task.
SW1
SW1(config-if)# int e3/1-2
SW1(config-if)# channel-group 101 mode active
SW1(config-if)# no shut
SW1(config-if)#
SW1(config-if)# int port-chan101
SW1(config-if)# sw mode trunk
SW1(config-if)# no shut
SW1(config-if)#
SW1(config-if)# int e3/5-6
SW1(config-if)# channel-group 102 mode active
SW1(config-if)# no shut
SW1(config-if)#
SW1(config-if)# int port-chan102
SW1(config-if)# sw mode trunk
SW1(config-if)# no shut
After configuring the port-channel on the first switch. It will come only, but the ports will be
in isolated state, because there is no LACP communication received back from the neighboring
switch.
SW1(config-if)# sh port-chan summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
101 Po101(SD) Eth LACP Eth3/1(I) Eth3/2(I)
102 Po102(SD) Eth LACP Eth3/5(I) Eth3/6(I)
SW1(config-if)#
This will change as soon as we start configuring the other 2 switches for port-channeling
the interfaces together.
SW2
SW2(config-if)# int e1/1-2
SW2(config-if)# channel-group 101 mode active
SW2(config-if)# no shut
SW2(config-if)#
SW2(config-if)# int port-chan101
SW2(config-if)# sw mode trunk
SW2(config-if)# no shut
SW2(config-if)#
SW2(config-if)# int e1/5-6
SW2(config-if)# channel-group 103 mode active
SW2(config-if)# no shut
SW2(config-if)#
SW2(config-if)# int port-chan103
SW2(config-if)# sw mode trunk
SW2(config-if)# no shut
SW3
SW3(config-if)# int e1/1-2
SW3(config-if)# channel-group 102 mode active
SW3(config-if)# no shut
SW3(config-if)#
SW3(config-if)# int port-chan102
SW3(config-if)# sw mode trunk
SW3(config-if)# no shut
SW3(config-if)#
SW3(config-if)# int e1/5-6
SW3(config-if)# channel-group 103 mode active
SW3(config-if)# no shut
SW3(config-if)#
SW3(config-if)# int port-chan103
SW3(config-if)# sw mode trunk
SW3(config-if)# no shut
Now when the port-channeling is checked we see that the interfaces are in participating
state which means that the port-channels are up and running.
SW1
SW1(config-if)# sh port-chan summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
101 Po101(SU) Eth LACP Eth3/1(P) Eth3/2(P)
102 Po102(SU) Eth LACP Eth3/5(P) Eth3/6(P)
SW1(config-if)#
SW2
SW2(config-if)# sh port-chan summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
101 Po102(SU) Eth LACP Eth1/1(P) Eth1/2(P)
103 Po103(SU) Eth LACP Eth1/5(P) Eth1/6(P)
SW2(config-if)#
SW3
SW3(config-if)# sh port-chan summ
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
102 Po102(SU) Eth LACP Eth1/1(P) Eth1/2(P)
103 Po103(SU) Eth LACP Eth1/5(P) Eth1/6(P)
SW3(config-if)#
Now we can move on to the first task of configuring the port-security feature.
This first question states that only 10 hosts may be connected to a certain ethernet interface on
SW2.
The action that should be taken when the maximum has been reached is that the interface
should be shutdown when another host is connected. Now this final point is an easy one, as by
default the action that is taken in case of a violation of the port-security rules is to shut
down the interface.
There are 3 actions that can be taken when a port-security rules are violated.
Shutdown
This action means that the interface will go in to errdisable state and the interface is
completely shutdown at that point. After it is re-enabled it keeps its port-security
configuration without changing anything. Including the previously learnt MAC addresses.
This is the default mode.
Restrict
Traffic from secure MAC addresses is allowed on the interface, but traffic from any
unsecure MAC addresses is dropped and a count is kept for the dropped packets
Protect
Traffic from secure MAC addresses is still allowed, but the interface is protected as MAC
address learning is disabled right after the first unsecure MAC address is seen. This
means that new MAC addresses are no longer learnt. Traffic from previously learned safe
MAC addresses can still pass through the interface
Let’s start with the configuration of the interface to protect it for 10 MAC addresses.
In this case we will statically configure the violation again, to ensure it’s done. Although it’s
a default. You will never get points taken off your score for over configuration that doesn’t
harm anything related to the tasks.
SW2
SW2(config)# feature port-security
SW2(config)# int e1/11
SW2(config-if)# switchport
SW2(config-if)# switchport port-security
SW2(config-if)# switchpor port-security maximum ?
<1-1025> Maximum addresses 1 to 1025
As is shown in the show run output. The violation is not configured in the configuration,
because it’s a default.
Do not forget to enable the port-security feature by entering ‘switchport port-
security’. Without this command, the other commands related to it, will show up in the
configuration, but the feature is not enabled on this interface and will not work.
version 5.1(3)N1(1)
interface Ethernet1/11
switchport port-security
switchport port-security maximum 10
With the following output you are able to check if it’s correct what you configured and what the
current status of the connection is.
SW2(config-if)# do sh port-security
Total Secured Mac Addresses in System (excluding one mac per port) : 0
Max Addresses limit in System (excluding one mac per port) : 8192
----------------------------------------------------------------------------
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
----------------------------------------------------------------------------
Ethernet1/11 10 0 0 Shutdown
============================================================================
SW2(config-if)#
The next question states that the entries that are learnt through this interface should time-out
after a certain time. By default when MAC addresses are learnt through the interface they are
kept in the database forever, as aging is disabled.
You can change this behavior by configuring aging. This can be done in two ways.
Absolute
The absolute aging removes a MAC address after a certain amount of time. After the
MAC address has been learnt, the switch starts the timer and after the configured
amount of time, it deletes the entry from the database.
Inactivity
After the last packet received from the MAC address. The switch starts a timer, when
after it expires the MAC address is deleted from the database. When a new packet is
received from the host, the timer is reset again.
The question states that the MAC address entry should be removed after inactivity for 6
minutes. This means we are going to use this option for aging under the interface.
SW2
SW2(config)# int e1/11
SW2(config-if)# switchport port-sec aging ?
time Port-security aging time
type Type of timers
After configuring the aging, you will see it is applied to the interface.
SW2(config-if)# sh port-sec int e1/11
Configured Port Security : Enabled
Opertional Port Security : Enabled
Port Status : Secure Down
Violation Mode : Shutdown
Aging Time : 6 mins
Aging Type : Inactivity
Maximum MAC Addresses : 10
Total MAC Addresses : 0
Configured MAC Addresses : 0
Sticky MAC Addresses : 0
Security violation count : 0
SW2(config-if)#
The next question asks to configure port-security on another interface with statically
configured MAC addresses. As soon as you configure the static MAC addresses you should not
forget to also put on the maximum number of MAC addresses. When you do not configure the
maximum amount, the interface will keep learning MAC addresses indefinitely.
SW3
SW3(config)# feature port-security
SW3(config)# int e1/11
SW3(config-if)# switchport
SW3(config-if)# switchport port-security
SW3(config-if)# switchport port-security maximum 5
SW3(config-if)# switchport port-security mac-address ?
E.E.E 48 bit mac address format HHHH.HHHH.HHHH (Option 1)
EE-EE-EE-EE-EE-EE 48 bit mac address format HHHH.HHHH.HHHH (Option 2)
EE:EE:EE:EE:EE:EE 48 bit mac address format HHHH.HHHH.HHHH (Option 3)
EEEE.EEEE.EEEE 48 bit mac address format HHHH.HHHH.HHHH (Option 4)
sticky Sticky MAC address
Do not forget to enable the port-security feature on the interface itself, otherwise it will
not work.
After configuring the 5 MAC addresses statically we verify that configuration is applied.
SW3(config-if)# show port-security
Total Secured Mac Addresses in System (excluding one mac per port) : 4
Max Addresses limit in System (excluding one mac per port) : 8188
----------------------------------------------------------------------------
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
----------------------------------------------------------------------------
Ethernet1/11 5 5 0 Shutdown
============================================================================
SW3(config-if)# show port-security int e1/11
Configured Port Security : Enabled
Opertional Port Security : Enabled
Port Status : Secure Down
Violation Mode : Shutdown
Aging Time : 0 mins
Aging Type : Absolute
Maximum MAC Addresses : 5
Total MAC Addresses : 5
Configured MAC Addresses : 5
Sticky MAC Addresses : 0
Security violation count : 0
You can see that the configured MAC addresses are equal to the maximum number of allowed
MAC addresses on this interface. Meaning that only these addresses are allowed on the
interface and no other.
Then we are asked to configure an action after violating packets are received. As previously
explained. This means that we will be using the ‘restrict’ action, as this is the only action
that will keep the interface up, but keeps a packet count of all violating packets. Which is what
the question asks for.
SW3
SW3(config)# int e1/11
SW3(config-if)# switchport port-security violation restrict
SW3(config-if)#
Next up is the configuration of the port-channel between SW2 and SW3. We should only allow
up to 100 MAC addresses on this interface before denying access to new ones and protecting
the interface by stopping the MAC address learning feature.
This means that we will be using the protect action after a violation of the port-security
rules.
When you read ahead on the questions, you see that the interface should automatically
‘remember’ all MAC addresses that are received. This feature basically means that we want all
MAC addresses received on the port-channel to be statically configured, except we don’t
want to configure them statically.
This feature is called ‘sticky’. When a MAC address is learnt and this feature is enabled, it is
automatically saved into the port-security database and never aged out.
We start with the configuration on the port-channel.
SW2
SW2(config-if)# interface po102
SW2(config-if)# switchport port-security
SW2(config-if)# switchport port-sec max 100
SW2(config-if)# switchport port-sec violation protect
Now we configured the basic security, but we also want the feature enabled to save the MAC
addresses automatically into the port-security database.
SW2
SW2(config-if)# switchport port-security mac-address sticky
SW3
SW3(config-if)# switchport port-security mac-address sticky
Now after the configuration we see that the maximum is correctly configured and that the
interface is already learning port-security entries.
SW3(config-if)# do sh port-security
Total Secured Mac Addresses in System (excluding one mac per port) : 4
Max Addresses limit in System (excluding one mac per port) : 8188
----------------------------------------------------------------------------
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
----------------------------------------------------------------------------
port-channel102 100 1 0 Protect
Ethernet1/11 5 5 0 Restrict
============================================================================
SW3(config-if)# sh port-security int po102
Configured Port Security : Enabled
Opertional Port Security : Enabled
Port Status : Secure UP
Violation Mode : Protect
Aging Time : 0 mins
Aging Type : Absolute
Maximum MAC Addresses : 100
Total MAC Addresses : 2
Configured MAC Addresses : 0
Sticky MAC Addresses : 2
Security violation count : 0
SW3(config-if)#
It’s possible to check which MAC addresses are learnt in the database. The entries are placed
into the running configuration. This means that they are not automatically saved after
reboots. This only happens after you save the configuration manually on the box.
SW3(config-if)# show port-security address
Total Secured Mac Addresses in System (excluding one mac per port) : 5
Max Addresses limit in System (excluding one mac per port) : 8187
----------------------------------------------------------------------
Secure Mac Address Table
----------------------------------------------------------------------
Vlan Mac Address Type Remaining Remotely Remotely Ports
age learnt aged
(mins) out
---- ----------- ------ ------ ------- ----- ----
1 BA01.DAD3.C0FF STATIC 0 No No Ethernet1/11
Finally we should ensure that the maximum of 100 MAC addresses is divided among a few
VLANs. The VLANS should get an even share, except for VLAN 10, which should get 66% of the
maximum (or 2/3).
Let’s create the VLANs and configure this maximum on both switches.
VLAN 10 will get 66 of the 100 MAC addresses. Which leaves 33 for even distribution among
the remaining 3 VLANS. This means a total of 11 MAC addresses per VLAN.
SW2
SW2(config-if)# vlan 10,11,12,13
SW2(config-vlan)# exit
SW2(config)# int port-channel102
SW2(config-if)# switchport port-sec maximum 66 vlan 10
SW2(config-if)# switchport port-sec maximum 11 vlan 11
SW2(config-if)# switchport port-sec maximum 11 vlan 12
SW2(config-if)# switchport port-sec maximum 11 vlan 13
ERROR: Maximum value on the port is less than the vlan value
SW2(config-if)# switchport port-sec maximum 10 vlan 13
SW2(config-if)#
Now because we can’t divide the number completely in whole numbers. We get an error
message that we configure too much MAC addresses per VLAN.
We solve this by giving the last VLAN one address less, as we can’t up the total amount of MAC
addresses, but we are still dividing the amount evenly as possible.
version 5.1(3)N1(1)
feature port-security
interface port-channel102
switchport port-security
switchport port-security maximum 100
switchport port-security violation protect
switchport port-security mac-address sticky
switchport port-security maximum 66 vlan 10
switchport port-security maximum 11 vlan 11
switchport port-security maximum 11 vlan 12
switchport port-security maximum 10 vlan 13
interface Ethernet1/11
switchport port-security
switchport port-security aging type inactivity
switchport port-security aging time 6
switchport port-security maximum 10
SW2(config-if)#
After the configuration we verify the configuration and check all our previous configuration.
The last 3 questions of this task are about the security of an interface on SW3. To simulate a
router we configure an interface on SW2 as a routed interface.
When we configure the SVI interface on SW3 for VLAN 100 and configure Ethernet1/7 as a
trunk interface. We should already be able to ping each other.
Do this before continuing with the port-security task so you are sure the Layer 3 path is
working correctly.
SW2
SW2(config)# int e1/7
SW2(config-if)# no switchport
SW2(config-if)# ip add 198.18.100.1/24
SW2(config-if)# no shut
SW2(config-if)# do sh run int e1/7
version 5.1(3)N1(1)
interface Ethernet1/17
no switchport
ip address 198.18.100.1/24
SW2(config-if)#
SW3
SW3(config-if)# vlan 100
SW3(config-vlan)# exit
SW3(config)# feature interface-vlan
SW3(config)# int vlan 100
SW3(config-if)# ip add 198.18.100.2/24
SW3(config-if)# no shut
SW3(config-if)# int e1/7
SW3(config-if)# switchport mode access
SW3(config-if)# switchport access vlan 100
SW3(config-if)# no shut
SW3(config-if)#
After the configuration of the Layer 3 routed interface on SW2 and the access port on SW3
where we are able to configure the port-security at a later stage. We should now be able
to ping the other end.
It might happen that this doesn’t work from the perspective of SW3.
SW3(config)# do ping 198.18.100.1
PING 198.18.100.1 (198.18.100.1): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
This might be related to the missing of a trunk allowed list on the port-channel between
the switches. This can cause it that the ping does not work.
SW3
SW3(config)# int po102
SW3(config-if)# sw trunk allowed vlan 10-13
After placing a trunk allowed list on the port-channel you are sure that the ping will work in
all cases.
SW3(config)# ping 198.18.100.1
PING 198.18.100.1 (198.18.100.1): 56 data bytes
64 bytes from 198.18.100.1: icmp_seq=0 ttl=254 time=3.167 ms
64 bytes from 198.18.100.1: icmp_seq=1 ttl=254 time=4.923 ms
64 bytes from 198.18.100.1: icmp_seq=2 ttl=254 time=4.962 ms
64 bytes from 198.18.100.1: icmp_seq=3 ttl=254 time=4.976 ms
64 bytes from 198.18.100.1: icmp_seq=4 ttl=254 time=6.528 ms
Now we can continue with configuring the port-security question, the final question of this
task.
We should configure port-security for a specific MAC address on the interface of SW3. The
problem is that SW2 has a different MAC address on its physical interface. So when we configure
this. The interface will go in errdisable state.
Let’s configure the port-security feature as the question is expecting it.
SW3
SW3(config)# int e1/7
SW3(config-if)# switchport port-secu
SW3(config-if)# switchport port-sec max 1
SW3(config-if)# switchport port-sec mac-address 1234.5678.abcd
SW3(config-if)#
Now we know for sure that the interface will be disabled, because the interface on SW2 is not
using this MAC address as its source.
SW3(config)# show port-sec int e1/17
Configured Port Security : Enabled
Opertional Port Security : Enabled
Port Status : Secure Down
Violation Mode : Shutdown
Aging Time : 0 mins
Aging Type : Absolute
Maximum MAC Addresses : 1
Total MAC Addresses : 1
Configured MAC Addresses : 1
Sticky MAC Addresses : 0
Security violation count : 0
SW3(config)# show port-sec
Total Secured Mac Addresses in System (excluding one mac per port) : 4
Max Addresses limit in System (excluding one mac per port) : 8188
----------------------------------------------------------------------------
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
----------------------------------------------------------------------------
port-channel102 100 0 0 Protect
Ethernet1/11 5 5 0 Restrict
Ethernet1/17 1 1 0 Shutdown
============================================================================
SW3(config)# show int e1/17
Ethernet1/17 is down
To ensure that the port-security feature works like expected, we need to change the MAC
address of the SW2 interface. Then the interface comes back online and the port-security
feature learns the correct MAC address.
SW2
SW2(config-if)# int e1/7
SW2(config-if)# shutdown
SW2(config-if)# mac-address 1234.5678.abcd
SW2(config-if)# no shutdown
Total Secured Mac Addresses in System (excluding one mac per port) : 4
Max Addresses limit in System (excluding one mac per port) : 8188
----------------------------------------------------------------------
Secure Mac Address Table
----------------------------------------------------------------------
Vlan Mac Address Type Remaining Remotely Remotely Ports
age learnt aged
(mins) out
---- ----------- ------ ------ ------- ----- ----
1 BA01.DAD3.C0FF STATIC 0 No No Ethernet1/11
1 0010.4431.A1B3 STATIC 0 No No Ethernet1/11
Now we finished Task 1 and we will continue with other various security features.
The DHCP snooping feature is enabled on VLAN 50 and the interface Ethernet3/10 is used
for receiving DHCP Offer messages by the DHCP server. Don’t forget to configure the port
within the VLAN as the question states that the DHCP server is directly connected to the switch.
So this needs to be an access interface in VLAN 50.
We are only required to configure SW1 as the question does not state anything about
configuring other switches for this feature as well.
The next question states that we should include the Option 82 information regarding the
VLAN and Port number into the DHCP packet.
This is done by enabling this feature. Typically you would expect this feature for DHCP Relay
configurations, but in this case it’s supported in DHCP snooping as well.
SW1
SW1(config)# ip dhcp snooping information option
We can verify that the features are enabled in the correct way as we want them to.
The next question asks to configure a verification. This verification can be performed by the
switch to ensure that the DHCP packet is not spoofed. This means that the switch checks if the
Source MAC address is equal to the MAC address specified within the DHCP packet.
SW1
SW1(config)# ip dhcp snooping verify mac-address
SW1(config)#
This concludes the configuration for the DHCP snooping feature. At this point the switch starts
building up a database of entries of clients that log-in through this switch.
Every time the DHCP server hands out a lease to a client, this information is stored into the
DHCP snooping database. This database now offers a perfect Layer 2 MAC Address to
Layer 3 IP address mapping on the switch.
This information is then leveraged by other features like the one we are configuring now.
Dynamic ARP Inspection (DAI) is a feature to really ensure you are safe against MAC
address spoofing attacks.
Dynamic ARP inspection takes care of a number of features.
Source MAC spoofing
The Source MAC address of an ARP request is checked against the DHCP snooping
database to ensure the request came from the correct interface
Optimal routing of packets
Because the DHCP snooping database knows where to find the ultimate destination of
the ARP request. It only sends it out on trusted interfaces or even where the
requested address is located
Checks for Valid ARP packets
The DAI process checks if ARP packets are send like they should be and verified that
there is no faulty information in the packet.
Now it’s also possible to create exceptions to this case. You can create an Access List and apply
that to the DAI process to ensure that ARP request sent for a specific range of IP addresses is
always allowed.
And finally logging is a very important topic. How long do you want to keep track of log
messages for this feature.
We start by configuring the initial question related to DAI. We want to ensure that VLAN 50 is
secured for ARP Spoofing, meaning we need to enable DAI on that VLAN.
There is one exception and that is that it should not perform these checks on the port-channel
interfaces going to the other switches.
Let’s configure these two questions.
SW1
SW1(config)# ip arp inspection vlan 50
SW1(config)# int port-chan101
SW1(config-if)# ip arp inspection ?
trust Configure trust state
Vlan : 50
-----------
Configuration : Enabled
Operation State : Active
DHCP logging options : Deny
SW1(config-if)# show ip arp inspec interfaces
Now we need to make an exception for this feature for the 198.18.50.0/28 subnet. As just
discussed it’s possible to let DAI ignore certain ARP packets that are related to a special ARP
Access List.
Next we need to ensure that the last 50 messages are logged, both for denies and accepts.
SW1
SW1(config)#
SW1(config)# ip arp inspect ?
log-buffer Log Buffer Configuration
validate Validate addresses
vlan Enable/Disable ARP Inspection on vlans
We first ensure that both permits and denies are logged, by configuring ‘all’ logging.
Then we configure the maximum amount of entries. By default this is set to 32 messages.
SW1(config)# show ip arp inspect vlan 50
Vlan : 50
-----------
Configuration : Enabled
Operation State : Active
DHCP logging options : All
SW1(config)# show ip arp inspect log
The final question related to Dynamic ARP inspection is a feature to extend the default
verification methods that it performs. You can enable some extended checks, like previously
mentioned. One of them is checking for a Valid IP address.
This means that the DAI feature will check for invalid or faulty IP addresses in ARP packets and
will drop these when it sees an invalid IP.
This feature is enabled with a single command.
SW1
SW1(config)# ip arp inspect validate ip
Now we finished the DAI configuration and we can check if all our settings are correctly applied
.
SW1(config)# show ip arp inspection vlan 50
Vlan : 50
-----------
Configuration : Enabled
Operation State : Active
DHCP logging options : All
SW1(config)# show ip arp inspect interfaces
Vlan : 50
-----------
ARP Req Forwarded = 0
ARP Res Forwarded = 0
ARP Req Dropped = 0
ARP Res Dropped = 0
DHCP Drops = 0
DHCP Permits = 0
IP Fails-ARP Req = 0
IP Fails-ARP Res = 0
SW1(config)# show ip arp inspect log
We then need to make an exception to interface Ethernet3/12 where a host with a statically
configured IP address is connected to.
SW1
SW1(config)# ip source ?
binding Static binding
Do not forget to enable the IP Source Guard feature on interface Ethernet3/12 as well,
because otherwise IP spoofing attacks are still allowed trough as the interface is not
protected!
The final 2 questions are about another feature related to IP Source Guard, but only
applicable to Layer 3 interfaces.
On Layer 3 interfaces it’s easier to check if a spoofing is done as you can check if the source IP
address is really coming from that particular IP subnet.
When this check is done, so the traffic should come from the subnet on which it is received on.
It’s called ‘strict’ Unicast Reverse Path Forwarding Check (uRPF).
Another mode of this same technology is that the switch/router is able to reach the source IP
address based on its routing table. This is called ‘loose’ mode.
We first need to configure the Layer 3 interface that we want to apply it to.
SW1
SW1(config-if)# feature interface-vlan
SW1(config)# int vlan 50
SW1(config-if)# ip add 198.18.50.1/24
SW1(config-if)# no shut
The mode which we need for our question is the Loose mode as we agree on reaching the
network in any way possible even including a default route.
We will configure the uRPF check on the VLAN 50 SVI interface.
SW1(config-if)# ip verify ?
unicast Unicast Reverse Path Forwarding
When ‘rx’ is given as the configuration command then the strict mode would be activated.
This means that traffic entering on a specific VLAN with IP subnet, it should be using a Source IP
address of this VLAN and Subnet.
Which completes Task 2. Now we can continue with Task 3 which is about configuring
Access Control Lists for both IP and MAC addresses.
Within NX-OS there is no distinction between these types of ACL’s and all ACL’s are using the
Extended features. Which means that you can use all variables possible.
There are 2 types of ACL’s that are tested in the CCIE Data Center lab. In the previous task we
already configured an ARP ACL. In this task we will be configuring IP Access Lists and MAC
Access Lists. The biggest difference between the 2 is that an IP ACL can be used at almost
any place in the switch to match certain IP traffic. MAC ACL’s are used for VLAN Access
Control Lists (VACL). They offer the option to limit certain Layer 2 frames based on MAC
addresses to pass through the switch or not. IP Access Lists can only match on IP
addresses.
Be very careful when reading the questions in these tasks. The tasks can be very difficult in
wording, so read very well in which direction the traffic is flowing. You also need to be prepared
for questions about common applications, of which you should know the port numbers and the
traffic characteristics.
The following types of traffic are able to be identified through MAC Access Lists:
• Source and/or Destination MAC address
• VLAN ID
• IP (version 4) Access Lists are able to match a lot more different types of
traffic characteristics.
• Source IP address
• Destination IP address
• Upper level protocol (TCP, UDP, EIGRP, GRE, OSPF, PIM, IP-IP, ESP, etc.)
• IGMP types
• DSCP values
• Packet length
• Fragmented packets
With all this variation of possible matching in the Access List, it gives you a lot of flexibility.
Be aware for lengthy Access Lists when a lot of different rules are applied.
It’s also possible to create Access Lists that are connected to a time range. This means
that the Access List is only applied during a specified time range. Which could be that Internet
traffic is only allowed during business hours, or any other rule.
One important topic to think about in real life deployments is Logging. When every match and
deny is logged, this could cause a lot of traffic to the Supervisor / Control-Plane as this might
give thousands of log messages every second.
In our lab scenario we will not have any traffic running on the network, so we are safe to
configure any feature we like.
We start by configuring our first IP Access Control List.
By default, when you are configuring an Access Control List. There is already one rule
inserted into this list, which is a rule that denies all traffic when no other rule has been
specified.
The rule is always present at the bottom of the ACL and is never shown. So keep in mind that all
traffic that is not matched by any other rule is denied to pass through.
It’s a common practice, especially in CCIE scenarios to insert a deny for all traffic at the bottom
of an ACL to ensure that you do not forget about it.
In the first question of the task it’s stated that we should secure VLAN 50 for traffic that should
not be there. This means we will be using an IP Access Control List.
Let’s start by configuring our ACL for VLAN 50, be sure to pick names which are easily
remembered and easy to type.
The first requirement is to create an entry for a host to be allowed to the VLAN. If you read
carefully you will see that the questions are all related to traffic going TO the VLAN that we
want to secure.
One very important decision to make when creating ACLs, is in which direction the ACL needs
to be applied. The ACL can be applied in inbound and outbound direction.
Inbound
The inbound direction will match the traffic that comes IN to the interface, from the
layer 2 domain / subnet. It comes IN to the switch via this interface and there the traffic
is matched
Outbound
The outbound direction will match traffic that moves OUT of the interface onto the
Layer 2 domain / IP subnet. This matches traffic going TO hosts on the subnet connected
to the interface
For this task in this first Access List configuration we will be applying the ACL in outbound
direction.
SW1
SW1(config)# ip access-list VLAN50_OUT
The first task is to allow a single host (source address) to send traffic to the VLAN. We could
apply this rule to ‘any’ destination address, but we need to be as specific as possible. So we will
configure the ACL with a wildcard mask which complies with the entire VLAN.
SW1
SW1(config-acl)# permit ip host 198.18.255.100 198.18.50.0 0.0.0.255
SW1(config-acl)#
Now the ACL is not applied to the interface yet, but we will wait for the application of the ACL
until we finished building all the rules that we are required to in this task.
Next question is that some Secure Web traffic, meaning SSL or HTTPS connections, is coming
from a certain range. We need to ensure that VLAN 50 receives this traffic.
We need to allow a /18 mask. This means in wildcard masks that 6 bits of the third octet
need to be a 1 which totals to 63.
SW1
SW1(config-acl)# permit tcp 198.18.128.0 0.0.63.255 eq 443 198.18.50.0 0.0.0.255 gt
1024
SW1(config-acl)#
The traffic is coming from servers, meaning that the source port will be Secure Web, as the
VLAN 50 will contain the clients that used this port as the Destination port.
The clients are using non-reserved ports, which means that they are using ports that are not
reserved for special cases (like HTTPS). The port range is all ports above 1024. Luckily NX-OS
foresees in this and there is a operator to match ports that are greater than a specific number
or range.
The next question will include a number of services that need to be allowed, but we cannot
configure them inside of the ACL. We need another option to configure applications that need
to be matched. This is found from a feature which is borrowed from the Cisco ASA firewalls. It’s
now possible to combine certain equal parameters and use them in a single Access Control
Entry (ACE). This is just for ease of configuration as the switch then automatically generates
various entries based on the grouping.
These combinations are called object-groups. They can be made for both IP addresses and
for Applications (port numbers). In this case we will need both, because we can only have two
entries in the ACL to solve the question.
We cannot use one entry for the first subnet and the second entry for the second subnet,
because we have both TCP and UDP applications in this question. It’s not possible to create a
TCP and a UDP match in a single entry, so therefore we will require 2 entries.
SW1(config)# object-group ip ?
address Address object group
port IP port object group (can be used in IPv4 and IPv6 access-lists)
SW1(config-ipaddr-ogroup)# 198.19.0.0/16
SW1(config-ipaddr-ogroup)# 198.18.192.0/24
SW1(config-ipaddr-ogroup)# exit
We first created the object-group related to the IP subnets that are matched. Instead of
wildcard masks it’s also possible in NX-OS to use the CIDR notation, making an ACL much easier
to read.
SW1
SW1(config)# object-group ip port TASK3_5_PORT
SW1(config-port-ogroup)# ?
<1-4294967295> Sequence number
eq Match only packets on a given port number
gt Match only packets with a greater port number
lt Match only packets with a lower port number
neq Match only packets not on a given port number
no Negate a command or set its defaults
range Match only packets in the range of port numbers
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-port-ogroup)# eq ?
<0-65535> Port number
SW1(config-port-ogroup)# eq 80
SW1(config-port-ogroup)# eq 53
SW1(config-port-ogroup)# eq 110
SW1(config-port-ogroup)# eq 25
SW1(config-port-ogroup)# exit
Next we configure our applications. Ensure you are aware of the port numbers for these
common services. It’s easy rememberable, because you will need these port numbers in real-
life as well while troubleshooting your network.
In this case don’t forget to add DNS to the group where TCP will be matched against, because
DNS uses both a TCP and a UDP port number.
SW1
SW1(config)# ip access-list VLAN50_OUT
SW1(config-acl)# permit tcp ?
A.B.C.D Source network address
A.B.C.D/LEN Source network prefix
addrgroup Source address group
any Any source address
host A single source host
The question prohibits us from doing that, which means we will need the second option and
that is to apply this IP Access List to the VLAN like a VACL (VLAN ACL).
Normally those VACL’s are used to match and deny Layer 2 traffic, but it’s also possible to
match based on IP ACLs within the VACL.
So we will be configuring a VACL to match our newly created IP ACL under. In this case it
doesn’t really matter like with IP ACLs in which direction it is applied, as the VACL is always
applied to ALL traffic of the VLAN, so both inbound and outbound traffic is matched against it.
SW1
SW1(config)# vlan access-map VLAN50
SW1(config-access-map)# ?
action Specify the action clause
match Specify the match clause
no Negate a command or set its defaults
statistics Enable per-entry statistics for the ACL
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-access-map)# match ?
ip Display IP information
ipv6 Display IPv6 information
mac MAC configuration commands
SW1(config-access-map)# match ip
ip ipv6
SW1(config-access-map)# match ip ?
address Match an access list
Traffic is matched in both directions. The action that is chosen here is to forward traffic. All
traffic that is not allowed to pass through is not matched. So will hit the default deny anything
rule.
Now we are done with our first IP Access List.
Next up is another question where we are not sure about all information.
We need to protect a certain host of which we do not know the IP address. Therefore we can
only allow the entire VLAN traffic, but we need to protect the host. We do know on which port
it is connected.
IP Access Lists can be applied at multiple places. One of the more common points is the
VLAN interface (SVI). We just used a VACL to match an IP ACL. The last option is to
configure the IP ACL under a physical Interface, so only traffic from that interface is matched
against it.
SW2
SW2(config)# ip access-list E1/15_IN
We create an ACL where we deny traffic towards a single host. The NX-OS knows a shortcut for
this in terms of the host keyword in ACLs. Again keep attention to the direction of the traffic. In
this case it’s a client accessing a server, which should be denied. Meaning that it’s the
destination port number which is the application port.
SW2(config-acl)# permit ip any any
Finally we should only be securing against the particular host. All other traffic should still be
passed through this interface. So we need to explicitly allow this.
Next step is to apply the ACL to the interface.
SW2
SW2(config)# int e1/15
SW2(config-if)# ip ?
access-group Specify access control for packets
port Port policy
SW2(config-if)# ip port ?
access-group Specify access control for packets
As the traffic is coming from the VLAN going towards the server, we need to apply this ACL in
inbound direction.
Be very careful with using this command, as there are 2 different ones. The ‘access-group’
variant is used for Layer 3 traffic hitting the interface when it would be in a ‘no switchport’
configuration. Only when the ‘port’ command is used in this case, the traffic is matched
against it while the port is in a normal switching mode.
Our next question is related to the security of the management interface.
You can apply a ACL for protecting the control-plane on 2 positions. The first is on the VTY.
This means that you are putting an ACL on the Telnet and SSH lines to protect others from
not connecting to the box via Telnet or SSH. This doesn’t protect all other protocols.
When you protect the management interface (mgmt0), you will be protecting all traffic,
assuming that all management traffic is using this interface.
In this question we are looking for this application of the protection for rogue devices.
Access-list configuration is pretty standard and we should be familiar already. Because we are
only allowed to create a single entry we will need to use a special command.
Within the ACL you can create a range command, where all the ports in the specified range are
allowed in the entry.
As Telnet and SSH are using connecting port numbers, we can easily use this range command
for our question.
SW1
SW1(config-if)# ip access-list MGMT
SW1(config-acl)# deny tcp 192.0.2.0/24 range ?
<0-65535> Port number
Remember that you need to protect against Telnet and SSH sessions that are being set-up
towards the switch, which means the position of the port numbers.
Now we can apply the ACL to the interface. Now it doesn’t need to be a port ACL, but a regular
ACL as the management interface is a Layer 3 interface.
SW1
SW1(config-acl)# int mgmt0
SW1(config-if)# ip ?
access-group Specify access control for packets
address Configure IP address on interface
arp Configure ARP parameters
directed-broadcast IP directed-broadcast
port-unreachable Enable sending ICMP port-unreachable
redirects Send ICMP Redirect messages
unreachables Enable sending ICMP unreachables (other than port-
unreachable)
Then finally don’t forget to apply this rule to all the switches as they all need to be protected.
SW2
SW2(config-if)# ip access-list MGMT
SW2(config-acl)# deny tcp 192.0.2.0/24 any range 22 23
SW2(config-acl)# permit ip any any
SW2(config-acl)#
SW2(config-acl)# int mgmt0
SW2(config-if)# ip access-group MGMT in
SW3
SW3(config-if)# ip access-list MGMT
SW3(config-acl)# deny tcp 192.0.2.0/24 any range 22 23
SW3(config-acl)# permit ip any any
SW3(config-acl)#
SW3(config-acl)# int mgmt0
SW3(config-if)# ip access-group MGMT in
The next question is related to a special feature of the Nexus platform. It is related to SPAN
sessions, but this session is related to ACLs in conjunction with SPAN / Monitor sessions.
This feature is called ACL Capture.
SW1
SW1(config)# hardware access-list ?
capture Configure ACL capture
lou LOU
resource Hardware resource
update Configure atomic/non-atomic update and default-result
This feature first needs to be enabled globally, before it will work. With enabling of ACL
Capture. You will disable all ACL logging.
The type command is very important for this configuration. A normal SPAN session cannot be
used.
SW1(config-acl-capture)# destination interface ethernet3/23
SW1(config-acl-capture)# no shutdown
^
% Invalid command at '^' marker.
For some reason, the normal command does not work in this case. You will need to use the
abbreviated version.
SW1(config-acl-capture)# no shut
SW1(config-acl-capture)# exit
Next step is to create the Access-List and to connect the SPAN session to the ACL.
SW1
SW1(config)# ip access-list SPAN
SW1(config-acl)# permit tcp any any ?
<CR>
ack Match on the ACK bit
capture Enable packet capture on this filter for session
dscp Match packets with given dscp value
eq Match only packets on a given port number
established Match established connections
fin Match on the FIN bit
fragments Check non-initial fragments
gt Match only packets with a greater port number
log Log matches against this entry
lt Match only packets with a lower port number
neq Match only packets not on a given port number
packet-length Match packets based on layer 3 packet length
portgroup Dst port group
precedence Match packets with given precedence value
psh Match on the PSH bit
range Match only packets in the range of port numbers
rst Match on the RST bit
syn Match on the SYN bit
time-range Specify a time range
urg Match on the URG bit
The session is connected to the entry that you want to be copied to the SPAN session. Don’t
forget to finish the configuration with a permit for all other traffic. As this ACL is just applied to
an interface. It would break the other traffic moving through the interface.
Finally the ACL should be applied to the interface. Again pay attention to if this is a Layer 2 or a
Layer 3 interface.
SW1
SW1(config-acl)# interface e3/22
SW1(config-if)# ip port access-group SPAN in
SW1(config-if)#
The next question is related to Layer 2 security. Which means that we should create a MAC
Access List for VLAN 50.
Pay close attention to how the question is formulated. The access-list should only match
traffic from the server farm. Meaning that the mentioned MAC addresses are used as source
addresses for the access list.
The MAC access-list is created in a very similar way as that IP ACLs are created.
SW1
SW1(config-mac-acl)# permit ?
E.E.E Source MAC address (Option 1)
EE-EE-EE-EE-EE-EE Source MAC address (Option 2)
EE:EE:EE:EE:EE:EE Source MAC address (Option 3)
EEEE.EEEE.EEEE Source MAC address (Option 4)
any Any source address
Wildcard masks work exactly the same as with IP Access Lists. In this case we chose a
relatively easy mask.
SW1
You can specify which traffic is allowed, but this is not necessary in our case.
SW1
SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff any
SW1(config-mac-acl)# exit
Then the access-list needs to be applied to a VACL. We already created this VACL for VLAN 50.
So ensure you do not override the previous settings.
VLAN Access Maps work in a similar way as Route-Maps do. So we can create additional
entries in the map. The first will be for the IP ACL and the second for the MAC ACL.
SW1
SW1(config)# vlan access-map VLAN50 ?
<CR>
<1-4294967295> Sequence number
After the second entry in the VACL is created, we can verify that it did not overrule any of the
previous configuration of the VACL.
The next question is that we should be gathering statistics about all entries. This is enabled
with a single command in each of the VACL sequences.
SW1
SW1(config)# vlan access-map VLAN50 10
SW1(config-access-map)# statistics
SW1(config-access-map)# vlan access-map VLAN50 20
SW1(config-access-map)# statistics
SW1(config-access-map)#
The final question of this task is related to COntrol Plane Policing (CoPP). Within the
Nexus platform the CoPP is already configured, unlike in IOS. Rate-limiters / Policers
are configured for all control-plane protocols and are in working.
They are using templates according to the features that are used on the switch. By default
these are good templates and chances are that you will never have to change them.
Therefore we do not focus on tweaking these here.
However, if you are using the box strongly for Layer 2 or Layer 3 purposes you will need to
change the template, so the routing protocols are given more space over the Layer 2 control
protocols.
Default CoPP Policy (copp-system-policy-default)
Scaled Layer 2 CoPP Policy (copp-system-policy-scaled-l2)
Scaled Layer 3 CoPP Policy (copp-system-policy-scaled-l3)
Customized CoPP Policy (copp-system-policy-customized)
The question states that we should be using optimizations for Layer 3 routing and we have a
template for that! This is exactly the change that we need to perform on both SW2 and SW3.
SW2
SW2(config-access-map)# control-plane
SW2(config-cp)# service-policy input copp-system-policy-scaled-l3
SW3
SW3(config-access-map)# control-plane
SW3(config-cp)# service-policy input copp-system-policy-scaled-l3
To verify that the command has been executed correctly, we can use the following show
output.
SW2(config-cp)# show copp status
Last Config Operation: service-policy input copp-system-policy-scaled-l3
Last Config Operation Timestamp: 01:21:43 UTC Nov 9 2012
Last Config Operation Status: Success
Policy-map attached to the control-plane: copp-system-policy-scaled-l3
SW2(config-cp)#
Now SW2 and SW3 are using a correct CoPP policy scaled for the primary use of Layer 3 Routing
on the box.
With this final question we finished the task and we will continue with Task 4 about AAA
services.
Based on open standards and used in all platforms from all vendors.
Based on UDP traffic, but including guarantees if requests were received or not.
TACACS+
Cisco proprietary protocol. However it does finds its integration in other vendor
products
Based on TCP traffic
There are not real differences in RADIUS or TACACS+, other than the type of traffic. As of
features they both offer equal features. Sometimes applications are using only RADIUS or
TACACS+, but overall they are equal in terms of features and functionality.
In the first few questions of this task we are configuring the AAA servers. We cannot use any
global configuration for that, which means we should be using ‘groups’ for server
configurations. The advantage of groups over global configuration is that groups can be used
in different places in the configuration. The groups also offer failover to other AAA servers
when the primary one failed.
First we should configure the RADIUS and TACACS+ servers while doing configuration in
groups.
SW1
SW1(config)# aaa group ?
server Configure aaa server group
Now we can’t configure a RADIUS server within a group before the server has been globally
configured. So we need to configure it in global configuration first.
SW1
SW1(config-radius)# exit
After configuring the global server we can configure the server inside the group. Where we
can also specify the source interface and source VRF that we want to use for RADIUS
requests.
Next we will configure the TACACS+ server configuration.
SW1
SW1(config)# taca?
^
% Invalid command at '^' marker.
?
^
% Invalid command at '^' marker.
SW1(config)# taca
^
% Invalid command at '^' marker.
SW1(config)# aaa group server ?
radius RADIUS server group name
By default only RADIUS configuration is enabled. For TACACS+ you need to enable the
feature first.
SW1
SW1(config)#
SW1(config)# feature tacacs+
SW1(config)#
SW1(config)# tacacs-server host 172.16.100.202 ?
<CR>
key TACACS+ shared secret
port TACACS+ server port
test Paremeters to send test packets
timeout TACACS+ server timeout period in seconds
Now we have configured our servers both for TACACS+ and RADIUS.
As there is no specification on which switch the configuration should be applied, we need to
configure it on all switches.
SW2
SW2(config)# radius-server host 172.16.100.201 key IPexpertAAA
SW2(config)# aaa group server radius RADIUS_SERVERS
SW2(config-radius)# server 172.16.100.201
SW2(config-radius)# use-vrf management
SW2(config-radius)# source-interface mgmt0
SW2(config-radius)# exit
SW2(config)# feature tacacs
SW2(config)# tacacs-server host 172.16.100.202 key IPexpertAAA
SW2(config)# aaa group server tacacs+ TACACS_SERVERS
SW2(config-tacacs+)# server 172.16.100.202
SW2(config-tacacs+)# use-vrf management
SW2(config-tacacs+)# source-interface mgmt0
SW3
SW3(config)# radius-server host 172.16.100.201 key IPexpertAAA
SW3(config)# aaa group server radius RADIUS_SERVERS
SW3(config-radius)# server 172.16.100.201
SW3(config-radius)# use-vrf management
SW3(config-radius)# source-interface mgmt0
SW3(config-radius)# exit
SW3(config)# feature tacacs
SW3(config)# tacacs-server host 172.16.100.202 key IPexpertAAA
SW3(config)# aaa group server tacacs+ TACACS_SERVERS
SW3(config-tacacs+)# server 172.16.100.202
SW3(config-tacacs+)# use-vrf management
SW3(config-tacacs+)# source-interface mgmt0
The next question is related to monitoring the RADIUS servers. Within NX-OS it’s possible to
pro-actively monitor the reach ability of the AAA servers, which prevents timeouts while logging
in.
First we need to configure the time-out values on the groups before it’s declared automatically
dead.
SW1
SW1(config)# aaa group server radius RADIUS_SERVERS
SW1(config-radius)# deadtime ?
<0-1440> Length of time, in minutes
SW1(config-radius)# deadtime 22
SW1(config-radius)# exit
SW2
SW2(config)# aaa group server radius RADIUS_SERVERS
SW2(config-radius)# deadtime 22
SW2(config-radius)# exit
SW2(config)# aaa group server tacacs+ TACACS_SERVERS
SW2(config-tacacs+)# deadtime 22
SW2(config-tacacs+)# exit
SW3
SW3(config)# aaa group server radius RADIUS_SERVERS
SW3(config-radius)# deadtime 22
SW3(config-radius)# exit
SW3(config)# aaa group server tacacs+ TACACS_SERVERS
SW3(config-tacacs+)# deadtime 22
SW3(config-tacacs+)# exit
Next we configure the monitoring. This is done by doing a simple AAA log-in to the server.
Which is why we need the username and password.
SW1
SW1(config)# radius-server ?
deadtime Duration for which non-reachable server is skipped
directed-request Enable direct authentication requests to server
host RADIUS server's DNS name or its IP address
key Global RADIUS server shared secret
retransmit Global RADIUS server retransmit count
test Parameters to send test packets
timeout Global RADIUS server timeout period in seconds
SW3
By default there is a lengthy time-out value for AAA requests. Which means that users need
to wait long before they can log-in using another AAA server or perform a fall-back to the local
database.
You are also able to configure the amount of retransmits that should be done.
Therefore we will configure a short time-out value for log-in requests.
SW1
SW1(config)# radius-server timeout 2
SW1(config)# tacacs-server timeout 2
SW1(config)#
SW2
SW2(config)# radius-server timeout 2
SW2(config)# tacacs-server timeout 2
SW2(config)#
SW3
SW3(config)# radius-server timeout 2
SW3(config)# tacacs-server timeout 2
SW3(config)#
In the next questions we will be applying the AAA configuration to the log-in process of the
switches.
Keep in mind that by default when you configure authentication, it only applies to the telnet
and SSH sessions. Console sessions are left out, as they are configured with a separate
command.
This is done, first of all to ensure you do not configure any faulty parameters causing you to be
locked out from the switch. Besides that it’s common that you configure the console
connection only with a local authentication, because usually you only perform console log-in,
when there is something wrong with the switch. Meaning that the chance is high that you are
also not able to reach the AAA servers anymore.
First we only want SW2 to authenticate to the RADIUS server and a fall-back to the local
database.
The local database should only be queried once the RADIUS server is dead and does not
respond. In case the RADIUS server replies with a reject message. It’s not falling back to the
local database. This is because it means that the user is not allowed to log-in to the device. So a
fall-back would violate the authentication rules.
Now when the fall-back would be done. The switch would automatically use the same
username and password as would’ve been typed in. First we configure RADIUS authentication
on SW2.
SW2
SW2(config)# aaa authentication ?
dhchap Configure methods for dhchap
login Configure methods for login
Now when we do not configure any second method the switch will automatically fall-back to
the local username/password database and will not re-ask for a password, because it’s the
default. In our case we configured the fallback to be sure enough.
Next up is the console authentication which was just mentioned to be done differently from the
SSH and Telnet log-in. As there is no specification of the switch. It should be configured on all.
SW1
SW1(config)# aaa authentication login console local
SW2
SW2(config)# aaa authentication login console local
SW3
SW3(config)# aaa authentication login console local
Next question relates to the configuration of SW3 for authentication. This is where we need to
use a Cisco Proprietary protocol, meaning that we will use TACACS+ for this host.
SW3
SW3(config)# aaa authentication login default group TACACS_SERVERS
Next up is that users without a role, should not be able to log-in to the device. Users will get
roles pushed through RADIUS and TACACS+ attribute configurations.
By default when users do not get this attribute pushed. They will be configured with the
‘network-operator’ or ‘vdc-operator’ role.
This can be disabled with a single command.
SW1
SW1(config)# no aaa user default-role
SW2
SW2(config)# no aaa user default-role
SW3
SW3(config)# no aaa user default-role
The next question is related to the notification for AAA servers if the servers are available.
By default users do not get to know how they have been logged in, of course for security
purposes. We can enable this feature.
SW1
SW1(config)# aaa authentication login ?
ascii-authentication Enable ascii authentication
chap CHAP authentication for login
console Configure console methods
default Configure default methods
error-enable Enable display of error message on login failures
mschap MSCHAP authentication for login
mschapv2 MSCHAP V2 authentication for login
SW2
SW2(config)# aaa authentication login error-enable
SW3
SW3(config)# aaa authentication login error-enable
This can be verified when logging into the switch using SSH. There it’s shown that the AAA
servers were not reachable for log-in and that the local database is used instead.
TerminalServer#ssh 172.16.100.1
Nexus 5000 Switch
Password:
The next question is related to how NX-OS stores usernames and passwords in it’s database.
By default it stores them with an MD5 hashed password in the database, but this can be even
stronger. By using the AES encryption algorithm.
This is known as ‘type 6’ passwords. Where MD5 hashes are known as ‘type 5’.
Before this feature can be enabled, you will need to configure a Master Key which is used for
encrypting the passwords.
There is no key specified, so you can use something you like.
SW1
SW1# key config-key ascii
New Master Key: IPexpert123
Retype Master Key: IPexpert123
We should be encrypting the currently configured usernames and passwords so they are also
using the new encryption mechanism.
SW1(config)# exit
SW1# encryption re-encrypt obfuscated
Finally, don’t forget to enable it on all the switches, because there was no specification given on
which switch it should be enabled.
SW2
SW2# key config-key ascii
New Master Key: IPexpert123switch2
Retype Master Key: IPexpert123switch2
SW2# conf t
SW2(config)# feature password encryption aes
SW2(config)# exit
SW2# encryption re-encrypt obfuscated
As the authentication is a local decision we can use any key we want on the switch. It’s even
recommended because this increases the security between the switches.
SW3
SW3# key config-key ascii
New Master Key: IPexpert123switch3
Retype Master Key: IPexpert123switch3
SW3# conf t
SW3(config)# feature password encryption aes
SW3(config)# exit
SW3# encryption re-encrypt obfuscated
Finally we should enable accounting towards the RADIUS server to ensure that all commands
that any user typed in will be logged.
SW2
SW2(config)# aaa accounting default group RADIUS_SERVERS
As previously mentioned. NX-OS uses another mechanism than Classical IOS for giving rights to
users. This means that all AAA servers are currently configured to support the IOS privilege
levels. These are levels between 0 and 15 for determining what users are able to do on the
switches.
The NX-OS device performs a default mapping for privilege levels received through TACACS+ to
default existing roles on the devices.
14 vdc-admin role
15 network-admin role
You can also configure privilege levels within NX-OS. This needs to be enabled to configure
the privileges connected to the usernames. It’s also possible to configure the privilege
levels as roles within the configuration.
To just enable backwards compatibility what we want, we just need to enable the feature.
SW3
SW3(config)# feature privilege
Next up is the forcing the users to configure strong passwords (or not).
A strong password consists out of the following characteristics.
Minimal of 8 characters long
Does not contain many consecutive characters (such as abcd)
Does not contain many repeating characters (such as aaabbb)
Does not contain dictionary words
Does not contain proper names
Contains both uppercase and lowercase characters
Contains at least 1 or more numbers
By default this strength check for passwords is enabled. When you disable and re-enable
the feature. It does not check for the current passwords to be configured with a strong
password or not. Keep this in mind in a production network.
In this case we should disable the check on SW3.
SW3
SW3(config)# no password ?
strength-check Strength check of password
The final question of this task is to configure a username which expires. After the expiration
date and time. The user will not be able to log-in anymore.
SW3
SW3(config)# username rick ?
<CR>
expire Expiry date for this user account(in YYYY-MM-DD format)
keypair Generate SSH User Keys
password Password for the user
priv-lvl Privilege level which the user is to be assigned to
role Role which the user is to be assigned to
ssh-cert-dn Update cert dn
sshkey Update ssh key for the user for ssh authentication
Now we finished Task 4 and we will continue with a feature not specifically mentioned on
the blueprint, but very likely to be tested as other things might be depending on it.
Besides that it is using the RADIUS configuration from this task to simulate authentications.
Task 5: 802.1X
This fifth task will be about configuring the IEEE 802.1X feature. This feature takes care of
ensuring the identity of users connecting to the network.
This is done by offering users a username/password dialog screen. Where they need to enter
there network credentials before the proper VLAN is configured on the switchport.
Without authentication users can not access the network!
This technology is actually related to LAN security and usually found in workgroup switches.
Within a datacenter environment this is usually not seen.
The code is implemented in NX-OS, but only available in the Nexus 7000. The Nexus 5000 has
no features for configuring this.
The items is not on the CCIE Data Center lab blueprint, but the chances are that it is treated in
the lab exam, because for the next task about Cisco TrustSec it is required.
Because the 802.1X feature is not fully integrated in the OS it doesn’t have all the features
that it would usually have in a LAN environment.
The feature will not be a large portion of the exam, which is why we will not focus a lot on the
feature here.
First we are required to configure some basic functionality of 802.1X.
We need to be sure that on some of the interfaces of SW1 users are authenticated before they
get access to the network. They should be authenticated against the RADIUS server which we
configured in the previous task.
SW1
SW1(config)# feature dot1x
SW1(config)# aaa authentication dot1x default group RADIUS_SERVERS
SW1(config)#
First we configure the RADIUS authentication that should be used, before any of the interface
configuration is done.
SW1
SW1(config)# int e3/25-31
SW1(config-if)# dot1x port-control auto
SW1(config-if)#
The interfaces are enabled with a single command. Relatively easy feature to enable, especially
because not all code is implemented on the Nexus 7000 platform. There is no such thing as
Guest VLANs or other advanced features.
The feature is purely used for authentication of the user against the network.
By default the 802.1X standard allows for a single host to be connected through the interface.
When there are multiple hosts connected through the same interface they are not allowed
access unless they authenticate.
This is disabled by default, so when we want this, we need to enable support for multiple
hosts.
SW1
SW1(config)# int e3/26-27
SW1(config-if)# dot1x host-mode multi-host
Re-authentication timers are given in seconds. In our case for 1 hour we need to specify
3600 seconds as re-authentication timer.
The next task is about giving access to a printer on our network which has no client for 802.1X
authentication. We still want the device to get access to the network in the correct VLAN, but
as the printer has no client software, it cannot do this by itself so we need to help in this
process.
What we can do is to use the MAC address of the client, in this case the printer, as the
authentication parameter towards RADIUS.
The RADIUS server will receive a request for authentication filled out with the clients MAC
address. We need to make sure an entry exists for this MAC address and then the port is still
authenticated.
This feature is by default disabled and can be enabled on a per interface level.
SW1
SW1(config)# int e3/31
SW1(config-if)# dot1x mac-auth-bypass
The next question states that a client is allowed to re-issue authentication up to 4 times. This
means that it is allowed to re-authenticate after rejection of the RADIUS Server for another 3
times before the interface is fully shutdown.
We can also configure this parameter both globally and per interface. We will configure it per
interface as well.
SW1
SW1(config)# int e3/25-31
SW1(config-if)# dot1x max-reauth-req 4
Finally we should ensure that all activity is logged via accounting towards the RADIUS server.
SW1
SW1(config)# dot1x radius-accounting
SW1(config)# aaa accounting dot1x default group RADIUS_SERVERS
With this last configuration we finished the 802.1X task. Next up is a true Data Center feature
called Cisco TrustSec.
Besides encrypting traffic and tagging traffic it can also be used to selectively add switches to
the network as they are authenticated. This works almost the same as Port Security in MDS
switches and Fibre Channel links. Switches send CHAP or EAP challenges to each other to
authenticate.
All the verification can also be performed by a RADIUS server. Be aware that only Cisco ACS
supports the TrustSec features at the time of this writing.
As we can not test the full features in this lab, because of limited available hardware and
software. We will focus on the authentication of the switches themselves. So a secure data
center network is created.
We first enable Cisco TrustSec and the authentication against the RADIUS server.
SW1
SW1(config)# feature dot1x
SW1(config)# feature cts
SW1(config)# cts device-id SW1 password SW1p@ssw0rd
When enabling the feature it’s required to have it configured with a local ID and password. We
will use the password mentioned later in this task for the other questions.
SW1
SW1(config)# aaa authentication cts default group RADIUS_SERVERS
SW1(config)# aaa authorization cts default group RADIUS_SERVERS
SW3
SW3(config)# feature cts
SW3(config)# cts device-id SW3 password P@ssw0rdSW3
SW3(config)# aaa authentication cts default group RADIUS_SERVERS
SW3(config)# aaa authorization cts default group RADIUS_SERVERS
Next we should enable Cisco TrustSec (CTS) on the interfaces that we enabled 802.1X
on.
SW1
SW1(config)# interface e3/25-31
SW1(config-if)# cts dot1x
SW1(config-if)# shut
SW1(config-if)# no shut
The enabling of CTS requires a flap of the interface before it will work. In our case we have no
hosts connected to the interface, so it doesn’t really matter. In a real life network the shutdown
and no shutdown of the interface is required for the CTS feature to work properly.
The last few questions of this chapter are about another feature of Cisco TrustSec, which is
the authentication of switches to ensure your datacenter network is only consisting of switches
that you expect. Connections between switches do not come online automatically without
configuring a proper authentication.
This feature is called the SGT Exchange Protocol (SXP). It is primarily used for giving out
the Security Tags to other switches, but can therefore also be used for authenticating switches.
This is where we are going to use it for.
It requires manual configuration of the authentication keys as we cannot use RADIUS for this
task. We also cannot use the mgmt0 interface for the SXP protocol, so we need to create SVI’s
which we are allowed to.
SW1
SW1(config)# vlan 99
SW1(config-vlan)# exit
SW1(config)# interface vlan 99
SW1(config-if)# ip address 198.18.99.1/24
SW1(config-if)# interface po101
SW1(config-if)# sw trunk allowed vlan add 99
SW1(config-if)# interface po102
SW1(config-if)# sw trunk allowed vlan add 99
SW1(config-if)# exit
SW1(config)#
SW2
SW2(config)# vlan 99
SW2(config-vlan)# exit
SW2(config)# interface vlan 99
SW2(config-if)# ip address 198.18.99.2/24
SW2(config-if)# interface po101
SW2(config-if)# sw trunk allowed vlan add 99
SW2(config-if)# interface po103
SW2(config-if)# sw trunk allowed vlan add 99
SW2(config-if)# exit
SW2(config)#
SW3
SW3(config)# vlan 99
SW3(config-vlan)# exit
SW3(config)# interface vlan 99
SW3(config-if)# ip address 198.18.99.3/24
SW3(config-if)# interface po102
SW3(config-if)# sw trunk allowed vlan add 99
SW3(config-if)# interface po103
SW3(config-if)# sw trunk allowed vlan add 99
SW3(config-if)# exit
SW3(config)#
Finally we can configure the authentication parameters of the switches to ensure the SXP
protocol will run successfully on all switches.
The switches will need to be statically configured as either speaker or listener switches.
Not at the same time they should be doing both to each other.
SW1
SW1(config)# feature cts
SW1(config)# cts role-based enforcement
SW1(config)# cts sxp enable
SW1(config)# cts sxp connection peer 198.18.99.2 password required SW2p@ssw0rd mode
listener
SW1(config)# cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode
listener
SW2
SW2(config)# cts role-based enforcement
SW2(config)# cts sxp enable
SW2(config)# cts sxp connection peer 198.18.99.1 password required SW1p@ssw0rd mode
speaker
SW2(config)# cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode
speaker
With this final configuration we completed the task and finished the chapter!
We prepared some configuration for the next chapter, so leave all configuration in place while
moving on to the next chapter about management features!
Chapter 9:
Management
Features
Chapter 9: Management Features is intended to let you be familiar with the Management features
which are available on the Nexus platform. You will be configuring Role Based Access Control (RBAC),
SNMP, Syslog, NetFlow, NTP and many more.
We highly recommend to create your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab. Multiple topology
drawings are available for this chapter.
General Rules
Try to diagram out the task. Draw your own connections the way you like it
Take a very close read of the tasks to ensure you don’t miss any points during grading!
Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular
chapter
Solutions
The model of RBAC is completely different from the Privilege level style like was used in
Classical IOS. In IOS users were connected to a privilege level, which is a number between 0
and 15, on which privileges are configured. Those privileges are configured in the form
of which commands are allowed to be executed by that user.
The possibilities with this type of configuration is limited and this is changed in NX-OS. Now
roles are configured instead of privilege levels. Inside of the role configuration it’s still
possible to configure commands that are allowed to be executed, but they offer much more
features.
One of the possibilities of role configurations is that you are able to allow features to be
individually configured. This means you do not need to configure all the related commands in
the configuration, but can allow a whole set of configuration commands with just a single line in
the configuration by allowing a specific feature.
Per feature you are able to specify whether the user is allowed to only verify the working or
also change the configuration.
This configuration can then even be applied only to interfaces and VRFs that are configured in
the role. When a user has certain interfaces and/or VRFs specified, it’s not able to touch any
configuration on other interfaces and in other VRFs.
The role configuration is specific to the VDC and never shared.
There are a number of default roles pre-configured on the NX-OS devices.
Role Privileges
network-admin Complete access to all commands and configuration items
network-operator Access to all show commands except show running-config, users can
never change any configuration
vdc-operator Access to all show commands except show running-config, users can
never change any configuration limited to a VDC
This means we have to do our user configuration in a later stage of the configuration. We first
need to configure the role itself and the limitations.
We do not need to enable a certain feature, like with all options in NX-OS. The RBAC feature is
by default enabled and can not be disabled.
As the question states. We are not able to configure features directly under the role
configuration. In NX-OS it’s possible to combine features together in so-called feature-
groups. These groups can then be used multiple times in different role configurations to
prevent overlapping configuration items.
SW1
SW1(config)# role feature-group ?
name Feature-group name
The feature names can be difficult to get to know while the CLI will not auto-complete you in
this case. It will tell you when a faulty feature is configured.
There is a list of features available through a show command which might make it a lot easier to
configure the feature names, although they are easy to guess as well!
SW1(config-role-featuregrp)# show role feature
aaa (AAA service related commands)
arp (ARP protocol related commands)
cdp (Cisco Discovery Protocol related commands)
l3vm (Layer 3 virtualization related commands)
ping (Network reachability test commands)
snmp (SNMP related commands)
radius (Radius configuration and show commands)
syslog (Syslog related commands)
tacacs (TACACS configuration and show commands)
Now we know enough to perform the configuration for the role of user1.
SW1
SW1(config-role-featuregrp)# feature vlan
SW1(config-role-featuregrp)# feature svi
SW1(config-role-featuregrp)# feature spanning-tree
SW1(config-role-featuregrp)# feature hsrp
We need to configure access to FHRP protocols. In this case this only means HSRP, as that’s the
only feature that we can configure. When we try to configure the other FHRP’s we are not
able to perform this configuration.
SW1(config-role-featuregrp)# feature vrrp
Invalid role feature name vrrp
SW1(config-role-featuregrp)# feature glbp
Invalid role feature name glbp
SW1(config-role-featuregrp)# exit
Then we can start configuring the role itself with the feature group.
SW1
SW1(config)# role ?
feature-group Configure role feature-group
name Enter the role name
SW1(config-role)# rule ?
<1-256> Enter the rule number
SW1(config-role)# rule 1 ?
deny Deny rule
permit Permit rule
The configuration is done by creating rules in the role. There you are able to specify the
read-only or read-write functionality and then the feature or feature-group that
should get access when connected to a username.
Next up is that the user should be limited to only configure a few interfaces.
This is done by first denying access to all interfaces and later specifying which interfaces should
have access.
SW1
SW1(config-role)# interface ?
policy Configure the interface policy for this role
SW1(config-role-interface)# permit ?
interface Enter the range of interfaces accessible the role
Just like with normal interfaces you are able to configure interface ranges under the
configuration. Which is applied accordingly.
SW1(config-role-interface)# ?
no Negate a command or set its defaults
permit Permit access to interfaces (applicable if interface policy is 'deny')
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-role-interface)# exit
SW1(config-role)# exit
Now finally after configuring the features, the role and the interface limitation we can
configure the user that is going to use this role.
SW1
SW1(config)# username user1 role USER1_ROLE password User1p@ssw0rd
SW1(config)#
Role: USER1_ROLE
Description: new role
vsan policy: permit (default)
Vlan policy: permit (default)
Interface policy: deny
Permitted interfaces:
Ethernet3/1-10
Vrf policy: permit (default)
-------------------------------------------------------------------
Rule Perm Type Scope Entity
-------------------------------------------------------------------
1 permit read-write feature-group USER1_FEATURES
The best test is of course to log-in to the switch with this new username and verify what you
are able to configure now.
TerminalServer#ssh -l user1 172.16.100.2
Nexus 5000 Switch
Password: User1p@ssw0rd
Bad terminal type: "xterm-256color". Will assume vt100.
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
SW1#
SW1#
SW1# ?
clear Reset functions
configure Enter configuration mode
debug Debugging functions
show Show running system information
end Go to exec mode
exit Exit from command interpreter
SW1# show ?
spanning-tree Show spanning tree information
vlan Vlan commands
SW1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)# ?
spanning-tree Spanning Tree Subsystem
vlan Vlan commands
end Go to exec mode
exit Exit from command interpreter
SW1(config)#
As is shown in the above outputs. The options for viewing and configuring items is very limited.
This limitation is enforced by the role configuration so our first questions are completed.
Next up is the configuration of another user with other limitations.
This user should again be limited to a number of features. Plus it’s only allowed to review
settings and execute show commands.
We are not allowed to create another feature-group or configure the features for all the
routing protocols. This means that we should find another creative solution.
When we verify the feature-group configuration. We see that there is a default feature-
group available on the switch in the OS.
SW1(config)# show role feature-group
ERROR: Feature "router-isis" does not exist
feature group: L3
router-bgp (Border Gateway Protocol related commands)
router-eigrp (Enhanced Interior Gateway Routing Protocol related commands)
router-ospf (Open Shortest Path First protocol related commands)
router-rip (Routing Information Protocol related commands)
As we need to configure the routing protocols we can use this default feature group for the
role configuration.
Just like with permitting only selected interfaces, we can also limit the configuration of certain
VRFs or VSANs. In this case we should limit the VRFs that the user is allowed to configure.
SW1
SW1(config-role)# vrf ?
policy Configure the vrf policy for this role
After configuring the username with the correct role assignment we finished the second
user’s role configuration.
The next user is only allowed to configure management protocols. This means that we should
create a feature-group consisting of all management protocols that we can find in the
overview and add those to the group together with the possibility to upgrade software of the
device.
SW1(config-role-featuregrp)# show role feature
aaa (AAA service related commands)
arp (ARP protocol related commands)
cdp (Cisco Discovery Protocol related commands)
l3vm (Layer 3 virtualization related commands)
ping (Network reachability test commands)
snmp (SNMP related commands)
radius (Radius configuration and show commands)
syslog (Syslog related commands)
tacacs (TACACS configuration and show commands)
install (Software install related commands)
license (License related commands)
callhome (Callhome configuration and show commands)
platform (Platform configuration and show commands)
access-list (IP access list related commands)
svi (Interface VLAN related commands)
hsrp (Hot Standby Router Protocol related commands)
igmp (Internet Group Management Protocol related commands)
msdp (Multicast Source Discovery Protocol related commands)
vlan (Virtual LAN related commands)
dot1x (DOT1X related commands)
ipfib (IP Forwarding Information Base related commands)
eth-span (Ethernet SPAN related commands)
router-bgp (Border Gateway Protocol related commands)
router-rip (Routing Information Protocol related commands)
ethanalyzer (Ethernet Analyzer)
router-ospf (Open Shortest Path First protocol related commands)
router-eigrp (Enhanced Interior Gateway Routing Protocol related commands)
spanning-tree (Spanning Tree protocol related commands)
acl (FC ACL related commands)
sfm (ISCSI flow related commands)
sme (Storage Media Encryption feature related commands)
fcns (Fibre Channel Name Server related commands)
fcsp (Fibre Channel Security Protocol related commands)
fdmi (FDMI related commands)
fspf (Fabric Shortest Path First protocol related commands)
rlir (Registered Link Incident Report related commands)
rscn (Registered State Change Notification related commands)
span (SPAN session relate commands)
vsan (VSAN configuration and show commands)
wwnm (WorldWide Name related commands)
zone (Zone related commands)
fcanalyzer (FC analyzer related commands)
sme-kmc-admin (SME commands authorized to kmc admin)
The marked lines show all the management protocols that we can allow access to.
SW1
SW1(config)# role feature-group name MGMT_PROTOCOLS
SW1(config-role-featuregrp)# feature aaa
SW1(config-role-featuregrp)# feature radius
SW1(config-role-featuregrp)# feature tacacs
SW1(config-role-featuregrp)# feature cdp
SW1(config-role-featuregrp)# feature snmp
SW1(config-role-featuregrp)# feature syslog
SW1(config-role-featuregrp)# feature callhome
SW1(config-role-featuregrp)#
Next up we configure the role and we add the final step, which is to allow software upgrades.
We could’ve also configured this together with the feature-group, but here we demonstrate
that we can both use a feature-group together with normal features in all types of
configuration.
SW1
SW1(config-role-featuregrp)# role name MAINTENANCE_ROLE
SW1(config-role)# rule 1 permit read-write feature-group MGMT_PROTOCOLS
SW1(config-role)# rule 2 permit read-write feature install
SW1(config-role)# exit
SW1(config)# username maintenance role MAINTENANCE_ROLE password MainTenanc3
SW1(config)#
Finally we create the username and we are done for this user. The installation of software
should of course be done with the install command, which is the only command the user has
access to.
The next user should only be allowed to configure Fibre Channel storage related features.
Again we check out the list of features and verify which should be added to a feature-group.
SW1(config-role-featuregrp)# show role feature
aaa (AAA service related commands)
arp (ARP protocol related commands)
cdp (Cisco Discovery Protocol related commands)
l3vm (Layer 3 virtualization related commands)
ping (Network reachability test commands)
snmp (SNMP related commands)
radius (Radius configuration and show commands)
syslog (Syslog related commands)
tacacs (TACACS configuration and show commands)
install (Software install related commands)
license (License related commands)
callhome (Callhome configuration and show commands)
platform (Platform configuration and show commands)
access-list (IP access list related commands)
svi (Interface VLAN related commands)
hsrp (Hot Standby Router Protocol related commands)
The marked lines show the storage related features that should be given access to. The final 3
features are related to a special storage feature, where dedicated users can only be given
access to these commands. They are not in scope of this configuration and are not necessary to
be configured.
SW1
SW1(config)# role feature-group name STORAGE
SW1(config-role-featuregrp)# feature sfm
SW1(config-role-featuregrp)# feature sme
SW1(config-role-featuregrp)# feature fcns
SW1(config-role-featuregrp)# feature fcsp
SW1(config-role-featuregrp)# feature fdmi
SW1(config-role-featuregrp)# feature fspf
SW1(config-role-featuregrp)# feature rlir
SW1(config-role-featuregrp)# feature rscn
SW1(config-role-featuregrp)# feature vsan
SW1(config-role-featuregrp)# feature wwnm
SW1(config-role-featuregrp)# feature zone
SW1(config-role-featuregrp)# feature fcanalyzer
SW1(config-role-featuregrp)#
SW1(config-role-featuregrp)# exit
SW1(config)#
SW1(config)#
SW1(config)# role name STORAGE_ROLE
SW1(config-role)# rule 1 permit read-write feature-group STORAGE
SW1(config-role)# exit
After configuring the role and the feature-group. We complete the configuration by creating
the username attached to the role.
SW1
SW1(config)# username storage-admin role STORAGE_ROLE password st0rage-@Dmin
SW1(config)#
The final role that should be configured is a NOC user. This user can have all rights like the one
of the ‘network-operator’, but is also allowed to issue Telnet and SSH commands from
the box.
Here we need to use command authorization and wildcards!
SW1
SW1(config)# role name NOC
SW1(config-role)# rule 1 permit command show
SW1(config-role)# rule 2 permit command telnet *
SW1(config-role)# rule 3 permit command ssh *
SW1(config-role)# exit
SW1(config)# username nocuser password NOCus3r role NOC
SW1(config)#
Password: NOCus3r
Bad terminal type: "xterm-256color". Will assume vt100.
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
SW1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
We are allowed to ensure configuration mode, but we cannot change anything in here.
SW1(config)# ?
no Negate a command or set its defaults
username Configure user information.
end Go to exec mode
exit Exit from command interpreter
SW1(config)# exit
SW1# show ?
aaa Show aaa information
access-lists List access lists
accounting Show accounting configuration
banner Show current motd banner message
SW1# telnet
SW1# ssh ?
WORD Enter hostname or user@hostname (Max Size 64)
SW1# ssh
SW2
SW2(config)# role distribute
SW3
SW3(config)# role distribute
The final step is to distribute all roles of SW1 to the switches by issuing a commit.
SW1
SW1(config)# role commit
2012 Nov 20 21:18:55 SW1 %CFS-2-MTS_REJECT: Verification failed reject MTS message
SAP 37:RR-token 0x3fe696b
Distribution: Enabled
Session State: Locking
SW1(config)#
The traffic is then copied to a destination interface. Multiple sources can be sent to the same
destination interface. Be aware that it’s possible to oversubscribe the destination interface with
configuring multiple sources.
There are ways to prevent that destination interfaces are oversubscribed, by configuring rate
limiting or truncation of packets.
All Nexus switches support up to 16 SPAN sessions to be configured. Only 2 can be active
simultaneous.
Virtual SPAN sessions are sessions that do not monitor a full interface, but only specific
VLANs on this physical interface.
Our configuration starts with configuring these Virtual SPAN sessions on the port-channels
connecting to SW2 and SW3.
The destination interface is specified. This can not be a port-channel member, only a physical
interface
SW1(config-monitor)# no shut
Finally we should configure the destination interface to support the SPAN traffic, which it by
default won’t.
We verify the working of the session and if we configured everything properly.
The interface will remain down as there is no cable connected, so the session will also show up
as being down, which is not a problem for our task as it is the expected result.
SW1(config-if)# do sh int e3/19
Ethernet3/19 is down (SFP not inserted)
Hardware: 10000 Ethernet, address: 0005.9b73.5752 (bia 0005.9b73.5752)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s
Beacon is turned off
Input flow-control is off, output flow-control is off
Switchport monitor is on
SW1(config-if)#
The next question is related to changing the MTU size of the monitoring traffic. This is done to
prevent the monitoring server for having too much traffic sent at too large MTU sizes.
The packet after the MTU size which is configured is deleted, so not the entire packet is sent to
the destination interface.
SW1
SW1(config)# monitor session 1
SW1(config-monitor)# mtu 1100
Now the MTU size will always be the same and never larger, not dependent on the source
packet.
The next question asks for another approach. The question asks for transportation of the traffic
across the network.
This technology is called Remote SPAN or RSPAN. It’s been around for a long time and it means
that traffic being monitored on one switch, is sent into a special dedicated VLAN which is then
picked up at another switch where the destination interface is located.
This is exactly what we need for this next question.
The traffic is being monitored on another switch and is sent to SW1 where the destination
interface is located.
SW1
SW1(config)# vlan 601
SW1(config-vlan)# remote-span
This is the trick for RSPAN sessions. The VLAN should be marked with the ‘remote-span’
keyword. Then the traffic is correctly picked up and received well.
SW1(config-vlan)# exit
SW1(config)# monitor session 2
SW1(config-monitor)# source vlan 601
SW1(config-monitor)# destination interface ethernet3/20
SW1(config-monitor)# no shut
The configuration of the RSPAN session is equal to a normal session for the rest. Keep in mind
to use another number for the session so you do not overlap with the previous question.
Now the traffic is being picked up and sent to the destination interface.
The next question is related to a technology introduced in the Nexus platform. Instead of
transporting traffic across a Layer 2 domain it’s now also possible to transport it across a
layer 3 domain inside a GRE tunnel.
This technology is called ERSPAN. It supports transport across a routed Layer 3 network.
The Nexus 5000 switches only support Source based sessions where as the Nexus 7000 also
supports destination sessions.
There is no specification on which IP addresses should be used for this session. So we will use
any routed network between the switches. For example VLAN 99 which is configured in a
previous chapter.
Then the configuration is pretty straight forward by just configuring the right IP addresses,
interfaces and some special values related to the ERSPAN session.
SW2
SW2(config)# monitor session 3 type erspan-source
The type of session is very important. Otherwise the ERSPAN configuration parameters could
not be configured.
SW2(config-monitor)# source interface ethernet1/17
SW2(config-monitor)# destination ip 198.18.99.1
SW2(config-monitor)# erspan-id 100
The ERSPAN ID is a unique identifier to ensure the ERSPAN session is not overlapped with
another session, which could cause mistakes in traffic monitoring.
SW2(config-monitor)# vrf default
SW2(config-monitor)# no shut
We should also configure the source address that the switch uses, as that is required for
configuration on the destination switch.
SW2(config)# monitor erspan origin ip-address 198.18.99.2
SW2(config-erspan-src)# exit
Then the destination interface should be marked as a SPAN destination port and we’re done!
The final tasks are related to tuning the ERSPAN configuration.
The first is about giving priority to ERSPAN traffic. This is done by giving the ERSPAN traffic a
higher DSCP bit, making it a higher priority traffic in Quality of Service configurations.
This configuration is done on the source.
SW2
SW2(config)# monitor session 3 type erspan-source
SW2(config-monitor)# ip dscp 11
Finally the granularity should be configured. This means in how much precision the traffic is
monitored. Configuration changes for this are done on the source as well.
SW2
SW2(config)# monitor erspan granularity ns
Task 3: NetFlow
In this task about NetFlow we will be configuring traffic monitoring based on the flow of
packets. This information is then used to form statistics about the network usage.
NetFlow works in a way that after it’s been enabled it will look for various packet streams that
have identical packet headers. Meaning they belong to a flow. This flow is a unidirectional
stream of packets. The flows are temporarily saved in the NetFlow cache en is then
exported to a Flow server where it can be used for analysis and accounting on network
traffic.
This process can take up a lot of resources. Which is why it’s done in hardware on the Nexus
platforms. NetFlow uses a sampling rate to capture traffic according to a configured rate. For
example every 1000 packets one is picked out and analyzed. The higher the sampling rate is,
the more precise the NetFlow information becomes, but also the required resources on the
device are getting higher.
There are several versions of the NetFlow export protocol. The Nexus switches support
version 5 and version 9. Where the highest version is recommended due to flexibility and
support for IPv6 and layer 2 information.
NetFlow support is very specific to the platform and the hardware. Currently only the Nexus
7000 supports the NetFlow feature.
The feature works by configuring various profiles. These profiles are bundled together
and applied to an interface. On this interface the specified information is then monitored.
We start by configuring the Nexus 7000 switch for the NetFlow record that the question
states.
The initial task is about creating a flow record based on IP address information. Together
with that the Flow ID and packets per second should be captured.
SW1
SW1(config)# feature netflow
SW1(config)# flow record IPX
SW1(config-flow-record)# ?
collect Specify a non-key field
description Provide a description for this Flow Record
match Specify a key field
no Negate a command or set its defaults
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-flow-record)# match ?
datalink Datalink (Layer 2) attributes
ip IP attributes
ipv4 IPv4 attributes
ipv6 IPv6 attributes
transport Transport layer fields
We first configure by which fields this record should be matched. Which are the IPv4
addresses like our question stated.
SW1
SW1(config-flow-record)# collect flow ?
sampler Sampler
As well as the Packets per Second counter. Be aware that you need to use 64-bit counters
as the question stated.
SW1(config-flow-record)#
In the next question it is asked that the information should be exported to a dedicated flow
server. Read ahead on the next question, because you will need some crucial information to
configure this export.
The last question of this task states that we should be able to export Layer 2 fields to the
Flow Server.
This automatically means that we need to use NetFlow version 9 in the Exporter
configuration.
SW1
SW1(config)# flow exporter IPX_EXPORT
SW1(config-flow-exporter)# version ?
5 Version 5 Export
9 Version 9 Export
SW1(config-flow-exporter)# version 9
SW1(config-flow-exporter-version-9)# ?
no Negate a command or set its defaults
option Version 9 Option Templates and Data
template Version 9 Template
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
SW1(config-flow-exporter-version-9)# exit
SW1(config-flow-exporter)# ?
description Provide a description for this Flow Exporter
destination Specify the destination address
dscp Optional DSCP
no Negate a command or set its defaults
source Source Interface for this destination
transport Transport Destination Port
version Specify the export version
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
After configuring the version and the destination we finished configuring our exporter.
Now the configuration hasn’t finished!
We continue with configuring the sampling rate of the packets.
SW1
SW1(config)# flow ?
exporter Define a Flow Exporter
monitor Define a Flow Monitor
record Define a Flow Record
timeout Define a Flow Timeout
SW1(config-flow-sampler)# mode ?
<1-64> Number of samples per sampling - 64 max
SW1(config-flow-sampler)# mode 5 ?
out-of M out of N packets
Now we configured our sampling profile for 5 out of 150 packets to be sampled.
Finally we should combine all this information together so the port-channels, like stated in the
first question, are monitoring flows and exporting them correctly.
SW1
SW1(config)# flow monitor IPX_MON
SW1(config-flow-monitor)# ?
description Provide a description for this Flow Monitor
exporter Add an Exporter to use to export records
no Negate a command or set its defaults
record Specify Flow Record to use
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
Within a monitor configuration, all items are combined together (exporter and record).
SW1
SW1(config)#
SW1(config)# int port-chan101
SW1(config-if)# ip flow monitor ?
WORD Name of Flow Monitor (Max Size 63)
SW1(config-if)# ip flow ?
monitor Apply a Flow Monitor to this interface
Finally the flow monitor should be configured under the interface where it’s necessary. Do
not forget to apply the sampler which we just configured!
SW1
There was no direction specified in which the flow monitor should be applied so we are going
to configure it on both directions.
SW1
SW1(config-if)# ip flow monitor IPX_MON output sampler IPX_SAMPLE
SW1(config-if)# int port-chan102
SW1(config-if)# ip flow monitor IPX_MON input sampler IPX_SAMPLE
SW1(config-if)# ip flow monitor IPX_MON output sampler IPX_SAMPLE
SW1(config-if)#
The final step is to apply the flow configuration to both directions and to both interfaces.
And after the application of the interfaces. We finished the configuration of NetFlow!
Of course there are many other combinations that can be configured, but for now we have
configured all properties, which should be enough for you to understand all variants that are
possible within the configuration.
These are no surprises as these are the management protocols which have been around for a
long time and are configured on many different platforms and software versions.
There is one important difference between IOS and NX-OS and that is that SNMP version 3 is
by default enabled on the device. This is done, because the Data Center Network
Manager (DCNM) uses SNMP v3 to communicate with the Nexus switches.
When you create a user-account the device automatically creates a SNMP version 3 user
account, making it possible to log-in using SNMPv3 as well.
Still NX-OS is compatible with SNMPv2c, because the entire world is still using it to manage
their networks.
We start by configuring the classical SNMP environment for our switch.
SW1
SW1(config)# snmp-server host ?
A.B.C.D|A:B::C:D|WORD IPv4 or IPv6 address or DNS Name of SNMP notification host
We also need to configure the version at which the traps should be sent. In this case we will be
using the Version 2c.
SW1
SW1(config)# snmp-server host 172.16.100.110 traps version 2c IPexpert
SW1(config)# snmp-server enable traps ?
<CR>
aaa Module notifications enable
bridge Enable SNMP STP Bridge MIB traps
callhome Module notifications enable
cfs Module notifications enable
config Module notifications enable
entity Module notifications enable
fcdomain Module notifications enable
fcns Module notifications enable
fcs Module notifications enable
fctrace Module notifications enable
fdmi Module notifications enable
feature-control Module notifications enable
fspf Module notifications enable
license Module notifications enable
link Module notifications enable
rf Module notifications enable
rmon Module notifications enable
rscn Module notifications enable
scsi Module notifications enable
snmp Module notifications enable
stpx Enable SNMP STPX MIB traps
sysmgr Module notifications enable
upgrade Module notifications enable
vsan Module notifications enable
vtp Module notifications enable
zone Module notifications enable
Before the switch will actually send traps. They need to be enabled first. By default the switch
will not send any traps, so then the question would not be answered correctly.
Ensure that traps are enabled when is stated in the question that a server needs to receive
traps. It’s not only the possibility, but it actually needs to receive them.
In this case we enable all the possible traps
The next question specifies that the server should also be able to access the switch via a specific
community string.
SW1
SW1(config)# snmp-server community ?
WORD SNMP community string (Max Size 32)
Read the question carefully as it clearly specified that the server should be able to ‘read’
information. So we can configure the read-only feature here.
The server is now able to access the switch to read out all information and it will receive traps.
The next questions are related to some contact information which is used in management
systems to identify devices based on their location. It’s a free text format, so you can fill out
anything you like.
SW1
SW1(config)# snmp-server location Utrecht, Netherlands, Europe
SW1(config)# snmp-server ?
aaa-user Set duration for which aaa-cached snmp user exists
community Set community string and access privs
contact Modify sysContact
context SNMP context to be mapped
enable Enable SNMP Traps
globalEnforcePriv Globally enforce privacy for all the users
host Specify hosts to receive SNMP notifications
location Modify sysLocation
mib Mib access parameters
protocol Snmp protocol operations
source-interface Source interface to be used for sending out SNMP notifications
tcp-session Enable one time authentication for snmp over tcp session.
user Define a user who can access the SNMP engine
As the next question states, we want the switch to enforce encryption for SNMPv3 traffic. One
of the biggest complaints about SNMPv1 and SNMPv2c is that it has no support for any
authentication other than the community strings and that it has no support for encryption.
SNMPv3 offers all of that, but it’s not required by default.
With a single command we can turn the switch so it will only accept encrypted SNMPv3
requests.
SW1
SW1(config)# snmp-server ?
aaa-user Set duration for which aaa-cached snmp user exists
community Set community string and access privs
contact Modify sysContact
context SNMP context to be mapped
enable Enable SNMP Traps
globalEnforcePriv Globally enforce privacy for all the users
host Specify hosts to receive SNMP notifications
location Modify sysLocation
mib Mib access parameters
protocol Snmp protocol operations
source-interface Source interface to be used for sending out SNMP notifications
tcp-session Enable one time authentication for snmp over tcp session.
user Define a user who can access the SNMP engine
We then should configure a user for SNMPv3 with rights. Now here is an overlap with the RBAC
configuration as SNMPv3 uses roles from RBAC to give access to what the management device
can access.
When you create a username, the switch will automatically create an SNMPv3 username with
the same rights as well. You can individually delete the SNMPv3 usernames when you like, but
it’s not possible to turn it off that the user is automatically created in the first place.
When an SNMPv3 user is created it also works vice versa. It will automatically create a user-
account as well.
SW1
SW1(config)# snmp-server user ?
WORD Name of the user (Max Size 32)
The next step is to configure the encryption key that is used. There is no problem in using the
same password for both keys. Which is also the default when a normal username is created.
SW1
Finally the user should be given the same rights as a currently existing user role. Now at first it
seems strange where to configure this, but the roles are called groups in this configuration.
You should use normal role names as the group name in this user configuration. After which
it will take over the rights the user has and work as the question stated.
Now we finished the part for configuring SNMP related configurations.
We can verify if everything is configured at is should.
SW1
SW1(config)# show snmp ?
<CR>
> Redirect it to a file
>> Redirect it to a file in append mode
community Show snmp community strings
context Show snmp context mapping entries
engineID Show snmp engineID
group Show snmp group
host Show snmp hosts
______________________________________________________________
NOTIFICATION TARGET USERS (configured for sending V3 Inform)
______________________________________________________________
Here it’s shown that all the users that have a regular user account are also configured as
SNMPv3 users.
SW1
Here you can verify which servers receive the traps that the switch sends.
Finally you can verify which SNMP communities are configured on the switch and how they are
operating.
The next management protocol which we will be configuring is the Syslog protocol. This
protocol is used for troubleshooting and debug messages and not so much for alarming.
For alarming SNMP traps are used and Syslog is used mostly during troubleshooting processes.
There are several levels of message priority you are able to see and which you are able to
process in separate log files when desired.
There are 7 levels of syslog priorities.
0: Emergency
1: Alert
2: Critical
3: Error
4: Warning
5: Notification
6: Informational
7: Debugging
All messages that the system can give have a priority attached. Usually up to level 4 is
shown as errors of the system. Level 5 and level 6 are messages related to processes
that perform actions. For example an OSPF neighbor comes online. When a neighbor goes
down, it’s possibly an error so this get’s a higher priority.
Level 7 is only used for debug messages and is therefore never shown. In IOS they were
always shown on the Console port, but this could cause serious issues as every console
character had impact on the CPU. Therefore in NX-OS this has been disabled and you need to
turn logging for debugging specifically on.
By default the following levels of logging are enabled. The logging monitor is the logging which
is shown in Telnet and SSH sessions.
SW1
SW1(config)# show logging
Currently it’s not even supported to raise the logging level of the console, without having
changed the console speed itself.
SW1(config)# logging console 6
Baud rate of console should be at least 38400 to increase logging level
The first Syslog question is so that Telnet and SSH sessions see informational level
messages. This is done by changing the logging level of monitor sessions.
SW1
SW1(config)# logging monitor ?
<CR>
<0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug
With the question mark it’s verifiable which level should be configured on the logging level.
SW1(config)# show logging
Now the informational messages are logged in Telnet and SSH sessions.
The next question states that debugging messages should be logged into a logfile.
SW1
SW1(config)# logging ?
abort Flushes cached data without committing and releases the lock
commit Commits cached data (of all msg types) and releases the lock
console Set console logging
distribute Enables/disables fabric distribution using cfs.
event Interface events
level Facility parameter for syslog messages
logfile Set File logging
message Interface events
module Set module logging
monitor Set terminal line(monitor) logging level
server Enable forwarding to Remote Syslog Server
timestamp Set logging timestamp granularity
Next we should be ensuring that the log files are marked with timestamps that are very
precise.
SW1
SW1(config)# logging timestamp ?
microseconds Timestamp in micro-seconds
milliseconds Timestamp in milli-seconds
seconds Timestamp in seconds (Default)
Microseconds is the most precise value we can use for our log messages. From now on the log
messages are logged with timestamps including a second, millisecond and microsecond value.
Finally we should configure a Syslog server which should receive all notifications that are
generated by the switch.
SW1
SW1(config)# logging ?
abort Flushes cached data without committing and releases the lock
commit Commits cached data (of all msg types) and releases the lock
console Set console logging
distribute Enables/disables fabric distribution using cfs.
event Interface events
level Facility parameter for syslog messages
logfile Set File logging
message Interface events
module Set module logging
monitor Set terminal line(monitor) logging level
server Enable forwarding to Remote Syslog Server
timestamp Set logging timestamp granularity
We should specify the correct level, which is 5 in this case where we want the
notifications to be forwarded to the server.
The last step is configuring the correct facility that should be used. Facilities in Syslog
are used for grouping messages together on the server itself and have no technical feature.
SW1
The master clock is now configured, so we can continue with the clients. We can use any
Layer 3 address active on the switch for this use. So we will be using the VLAN 99 IP
addresses.
SW2
SW2(config)# ntp server 198.18.99.1
SW3
SW3(config)# ntp server 198.18.99.1
After the configuration we can verify that the time is synchronizing against the master clock.
Keep in mind that this can take a long time sometimes. During the CCIE Lab it’s even better to
move on to the next task as you might need to wait up to 10 minutes before the NTP has been
synchronized correctly.
We should now protect SW1 against unwanted NTP clients to be synchronizing time with us.
SW1
SW1(config)# ip access-list NTP_CLIENTS
SW1(config-acl)# permit ?
<0-255> A protocol number
ahp Authentication header protocol
eigrp Cisco's EIGRP routing protocol
esp Encapsulation security payload
gre Cisco's GRE tunneling
icmp Internet Control Message Protocol
igmp Internet Group Management Protocol
ip Any IP protocol
nos KA9Q NOS compatible IP over IP tunneling
ospf OSPF routing protocol
pcp Payload compression protocol
pim Protocol independent multicast
tcp Transmission Control Protocol
udp User Datagram Protocol
The serve keyword enables the NTP server to filter out requests from various connecting NTP
systems. When serve is used, it will only accept requests and will never synchronize its own
time against the specified IP addresses in the ACL.
To protect the NTP process even more, the requests should be authenticated!
SW1
SW1(config)# ntp authentication-key 1 md5 TimeIPX
SW1(config)# ntp trusted-key 1
SW1(config)# ntp authenticate
We first configure the master clock for authentication. Do not forget to enable the
authentication process itself, otherwise NTP will continue without authenticating anything.
The clients should be configured a little different, because they should authenticate against the
server.
SW2
SW2(config)# ntp authentication-key 1 md5 TimeIPX
SW2(config)# ntp server 198.18.99.1 key 1
SW2(config)# ntp authenticate
SW3
SW3(config)# ntp authentication-key 1 md5 TimeIPX
SW3(config)# ntp server 198.18.99.1 key 1
SW3(config)# ntp authenticate
SW2
SW2(config)# clock timezone CET 1 0
SW2(config)#
SW3
SW3(config)# clock timezone CET 1 0
SW3(config)#
You first configure a name which identifies the timezone that you are in. Then you specify the
time offset in hours and minutes that is different from GMT time.
The final questions in this management task are related to a protocol proprietary to Cisco. This
protocol is known as the Cisco Discovery Protocol.
It is used to exchange information about the Cisco devices which are connected to the
interfaces. It’s extremely useful in your network environment as you immediately know what is
connected behind an interface and also at which interface on that other device. Additionally
you can even see what kind of device it is and when possible the management IP address to
connect to.
It’s also extremely dangerous when used in environments where a lot of third parties connect,
like Service Provider environments or Multi-Tenant Data Centers.
We should first set SW1 to identify itself via its serial number compared to its hostname
which is the default.
SW1
SW1(config)# cdp ?
advertise Highest CDP version supported on the switch
enable Enable/disable CDP on all interfaces
format Device ID format for CDP
holdtime CDP hold time advertised (in seconds)
timer CDP refresh time interval (in seconds)
This is of course totally unusable in a production world, but in this case it’s what the question
asks for.
Next we should configure that switches should not use the default 60 second advertisements
of CDP packets, but should change this to 10 second intervals.
SW1
SW1(config)# cdp timer 10
SW1(config)#
SW2
SW2(config)# cdp timer 10
SW2(config)#
SW3
SW3(config)# cdp timer 10
SW3(config)#
And the last question is that we want to disable CDP to be enabled on certain interfaces that
are connected to a third party.
SW2
SW2(config)# int e1/10-20
SW2(config-if-range)# no cdp ?
enable Enable/disable CDP on the interface
SW3
SW3(config)# int e1/10-20
SW3(config-if-range)# no cdp enable
SW3(config-if-range)#
If you want to disable CDP to run on the entire device you should globally disable it with the
command ‘no cdp run’.
After the checkpoint has been generated it’s possible to compare differences with the other
configuration files.
SW1
SW1# show diff ?
rollback-patch Show rollback patch between configuration files or checkpoints
The available rollbacks are shown which you can use for the comparing.
Then you should specify to which configuration the compare should be done. In this case we
are using the startup-config.
cdp enable
exit
interface Ethernet1/19
cdp enable
exit
interface Ethernet1/18
cdp enable
exit
interface Ethernet1/17
cdp enable
exit
interface Ethernet1/16
cdp enable
exit
interface Ethernet1/15
cdp enable
exit
interface Ethernet1/14
cdp enable
exit
interface Ethernet1/13
cdp enable
exit
interface Ethernet1/12
no switchport monitor
cdp enable
exit
interface Ethernet1/11
no switchport monitor
no switchport mode trunk
cdp enable
exit
interface Ethernet1/10
cdp enable
exit
no interface port-channel1011
no interface port-channel110
no cdp format device-id serial-number
no cdp timer 10
no vlan 601
no ntp trusted-key 1
no ntp authentication-key 1 md5 WeqjAPS 7
no ntp authenticate
no ntp server 198.18.99.1
no snmp-server community IPexpert group network-operator
no snmp-server host 172.16.100.110 traps version 2c IPexpert
no snmp-server globalEnforcePriv
no snmp-server location Utrecht, Netherlands, Europe
no snmp-server contact Rick Mur, IPexpert
no ip access-list NTP_CLIENTS
no username version3 role storage-admin
no username version3
no username readonly
no role name storage-admin
!
interface vfc1
switchport mode E
no shutdown
!
logging monitor
SW1#
The configuration output that is shown then includes a patch that you can use as a
configuration template. The configuration is made so when it is applied to the current
configuration (the one you are comparing with). Then it would rollback to that configuration.
It’s a very useful feature before doing changes, to automatically create your rollback script.
The script also ensures that commands are typed in the correct order, so you can easily copy
and paste it to the device when there are problems with your change or your current
configuration.
The next questions states that the configuration should be saved to a TFTP server on a weekly
basis on Sunday night at 10PM or 22:00.
The file-name should include some variables like the hostname and the date and time.
For this feature we are going to use a simple copy command, but we need to schedule it.
NX-OS knows a scheduler feature in which you are able to schedule CLI commands and also
use different variables to ensure the command executed includes things like the hostname
or the date and time.
We configure the scheduler job first, containing the commands that need to be executed.
SW1
SW1(config)# feature scheduler
SW1(config)# scheduler ?
aaa-authentication Password for AAA authentication(of logged in user)
job Define a job
logfile Scheduler log file configuration
schedule Define a schedule
transport Configure transport related configuration
Within the scheduler job you have access to the full CLI. This CLI access gives you the
possibility to change anything in the configuration or any show commands necessary.
Ensure you are using the correct variables in the job. These can be found in the Cisco
Documentation. For now the hostname and timestamp are the only variables available in the
scheduler, but new ones can be added in newer versions of software.
SW1
SW1(config)# scheduler schedule name CONFIG_BACKUP_SCHED
SW1(config-schedule)# job name SAVE_CONFIG
SW1(config-schedule)# ?
email-addr Add email addr to send output of jobs configured for this schedule
job Assign a job to the schedule
no Negate a command or set its defaults
time Specify a schedule to run jobs
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-schedule)# time ?
daily Specify a daily schedule
monthly Specify a monthly schedule
start Specify a future time schedule
weekly Specify a weekly schedule
After the creation of the job, a scheduler needs to be created. This finalizes the
configuration and ensures that the selected job is run at the given time and date.
We can verify if the job was executed successfully and what results where. When errors occur
during the execution of scheduler jobs. These are logged in a logfile.
It’s also possible to configure a username and password which is used for executing the jobs so
the accounting records are filled as well.
By default the admin user is used as the user to execute the jobs.
==============================================================================
N7K11-SW1(config)# show scheduler schedule
Schedule Name : CONFIG_BACKUP_SCHED
-----------------------------------------
User Name : admin
Schedule Type : Run on every Sunday at 22 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
Job Name Last Execution Status
-----------------------------------------------
SAVE_CONFIG -NA-
==============================================================================
SW1(config)#
Now the schedule job is scheduled and the configuration is backed up to a TFTP server every
week.
The next question is related to a banner that should be shown to users when they log in. As
with classical IOS. This is done using the Message Of The Day (MOTD) banner.
SW1
SW1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)# banner ?
motd Configure banner motd message
After configuring the banner it is shown when users are logging in to the device.
The last questions are related to the archiving features of the NX-OS platform. You are able to
save files and compress them to GZIP files. After which they can then be combined to TAR files.
This is especially good to do with tech-support files, as they are very lengthy and can
generate megabytes of clear text.
In the first question we need to save the tech-support and create a zip file manually.
SW1# show tech-support > bootflash:techsupport.txt
SW1# dir bootflash:
18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2
1256 Mar 18 16:33:53 2011 aaa_cnv.log
364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log
505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic
49152 Mar 18 16:14:02 2011 lost+found/
3785 Nov 07 20:01:14 2012 mts.log
21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin
25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin
25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin
181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin
156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin
188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin
4939579 Nov 23 12:55:41 2012 techsupport.txt
4096 Jan 01 01:02:43 2005 vdc_2/
After the output of the tech-support is saved to a text file on the flash. We see that the
size is almost 5 megabytes. Now we need to compress it.
SW1# gzip bootflash:techsupport.txt
SW1# dir bootflash:
18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2
1256 Mar 18 16:33:53 2011 aaa_cnv.log
364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log
505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic
49152 Mar 18 16:14:02 2011 lost+found/
3785 Nov 07 20:01:14 2012 mts.log
21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin
25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin
25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin
181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin
156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin
188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin
347146 Nov 23 12:55:41 2012 techsupport.txt.gz
4096 Jan 01 01:02:43 2005 vdc_2/
4096 Jan 01 01:02:43 2005 vdc_3/
4096 Jan 01 01:02:43 2005 vdc_4/
With the redirection-mode configuration (the > command) we can set how files should be
saved. By default they are saved as standard ASCII files, but now we changed it to ZIP files.
While the file is still saved as .txt file. The output is unreadable as it is zipped.
SW1# dir
18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2
1256 Mar 18 16:33:53 2011 aaa_cnv.log
364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log
3101 Nov 23 12:59:27 2012 interface.txt
505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic
49152 Mar 18 16:14:02 2011 lost+found/
3785 Nov 07 20:01:14 2012 mts.log
21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin
25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin
25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin
181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin
156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin
188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin
347146 Nov 23 12:55:41 2012 techsupport.txt.gz
4096 Jan 01 01:02:43 2005 vdc_2/
4096 Jan 01 01:02:43 2005 vdc_3/
4096 Jan 01 01:02:43 2005 vdc_4/
??x???+Y????
sj?8w?.?l??f????vٴHN}??n?I?? ڱ#x;v۶?w?Z?jyZ?7
? O| 2 #
?i??4??y?!
+?#?%?A?\?!]Pc;~?@
???i{Z?ۻO`K?K?5?�|?>?`U??U ?;??Pdk:2?F6??ˑڄF?????>w ???R|l?}??|z?zK?\?h{K?
&??zK4S?p?
?&??z??(i,????YI?(-a?F
?ҶoKKì&???-??b???????
ÿ?oh>
F-
????qΊ?}??{?;?lpOl,?f??????ν??? ?ڌyi??&@??????Z?8?-???v?{?k??
???退?^?Fz?n`t۟?3Z???`??)?ջ ?\KΥT??Iо?T?ϥ`??B??Ń?
?1???WJ???????Z?ݝw??*\|x???ɔ????+???.}??~I/???/S?8?Ee?V?+??z??k#??+?ş???:?? ߢ9v?v??
-
?Vc?&?fLӡ?[??m??J+??5?H?[u.?T?R3#?;??ٶ
?Q?j̀
???U[B1j?g?
Z}?Zo???g?=%u,?j??s
SW1# ?R?
We first specify the name of the archive and then specify which files should be included in
the archive.
SW1#
SW1# dir
18623 Nov 23 11:24:39 2012 CONFIG_BACKUP_2
1256 Mar 18 16:33:53 2011 aaa_cnv.log
364 Mar 18 16:33:53 2011 assoc_mgr_cnv.log
330925 Nov 23 13:05:32 2012 combined.tar.gz
3101 Nov 23 12:59:27 2012 interface.txt
505 Mar 21 17:26:12 2010 license_SSI140806X2_6.lic
49152 Mar 18 16:14:02 2011 lost+found/
3785 Nov 07 20:01:14 2012 mts.log
21778944 May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin
25164288 Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin
25152512 Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin
181095489 May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin
156932426 Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin
188758273 Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin
347146 Nov 23 12:55:41 2012 techsupport.txt.gz
4096 Jan 01 01:02:43 2005 vdc_2/
4096 Jan 01 01:02:43 2005 vdc_3/
4096 Jan 01 01:02:43 2005 vdc_4/
We finally see the file is created and that the archive contains the correct files just like we
wanted.
Just like the question specified, the TAR file should be compressed. Which is the default when
not specifying anything about the archive in the CLI.
This finished the task about the system management!
SW2
SW2(config)# diagnostic bootup level complete
SW3
SW3(config)# diagnostic bootup level complete
The results can be made visible with a show diagnostic results command.
Next we can start by configuring the Call Home feature.
We do not need to enable the feature, because it’s always enabled.
SW1
SW1(config)# feature call
^
% Invalid command at '^' marker.
SW1(config)#
SW1(config)#
SW1(config)# snmp-server contact Rick Mur, IPexpert
SW1
SW1(config)# callhome
SW1(config-callhome)# ?
alert-group Alert group
contract-id Service contract id of the customer
customer-id Customer id
destination-profile Configure destination profiles
duplicate-message Configure throttling of duplicate callhome alert messages
email-contact Email address of the contact person
enable Enable callhome
no Negate a command or set its defaults
periodic-inventory Configure periodic software inventory message dispatch
phone-contact Contact person's phone number
site-id Site id of the network where switch is deployed
streetaddress Configure replacement part shipping address.
switch-priority Priority of the switch(0-highest 7-lowest)
transport Configure transport related configuration
end Go to exec mode
exit Exit from command interpreter
pop Pop mode from stack or restore from name
push Push current mode to stack or save it under name
where Shows the cli context you are in
SW1(config-callhome)# alert-group ?
Configuration Events related to Configuration
Diagnostic Events related to Diagnostic
Environmental Power,fan,temperature related events
Inventory Inventory status events
License Events related to licensing
Linecard-Hardware Linecard related events
Supervisor-Hardware Supervisor related events
Syslog-group-port Events related to syslog messages filed by port manager
System Software related events
Test User generated test events
Now the destination e-mail address and method are specified. Next up is configuring the
source.
SW1
SW1(config-callhome)# email-contact ?
WORD Provide email address, example: jdow@xyz.com (Max Size 255)
The street address should also be specified as that is used as a field in the Callhome message.
SW1
SW1(config-callhome)# transport ?
email Configure email transport related configuration
We configure the SMTP server and the source e-mail address that should be used in our
configuration. Pay attention that you are only able to configure 1 source address and SMTP
server, but multiple destination addresses. The destination address is configured under the
destination-profile.
SW1
SW1(config)# ip name-server 172.16.100.111
Cannot write to the dns configuration -DNS is disabled
SW1(config)# ip domain-lookup
SW1(config)# ip name-server 172.16.100.111
Because we are configuring the name of the SMTP server in our configuration, we should use
DNS name resolution to ensure that the IP address of the mail server is resolved.
SW1(config-callhome)#
SW1(config-callhome)#
SW1(config-callhome)# destination-profile CRITICAL_MAILS ?
<CR>
alert-group Add alert group
email-addr Add email addr
format Callhome message format (default XML)
http Add http or https url
message-level Callhome message level(0-lowest urgency, 9-highest urgency)
message-size Configure maximum message size (default 2500000)
transport-method Callhome message sending transport-method
Finally we should specify the maximum message size. We want to sent all messages of any
urgency therefore we configured the level 0 urgency.
Next we should configure periodic inventory reports, which is another feature of CallHome.
SW1
SW1(config-callhome)#
SW1(config-callhome)# periodic-inventory ?
notification Enable periodic software inventory message dispatch
The next question specifies that SW1 is a very important switch which should be noticed in the
CallHome configuration. This is specified through the priority configuration.
SW1
SW1(config-callhome)# switch-priority ?
<0-7> Priority of the switch(0-highest 7-lowest)
SW1(config-callhome)# switch-priority 0
SW1(config-callhome)#
Finally we should ensure that Cisco TAC receives information, both via e-mail and HTTP
requests.
This is configured with XML destination-profiles. However we can only configure one
method per destination-profile. We can only configure one additional profile, so we
need to use one predefined profile.
The pre-defined profiles are configured with a set of alert-groups, but also give you room
for additional configuration, like the transport-methods.
We first configure the HTTP version using the pre-defined Cisco TAC profile.
SW1
SW1(config-callhome)# callhome
SW1(config-callhome)# destination-profile ?
CiscoTAC-1 Configure destination profile for XML message
WORD User defined destination profile name (Max Size 31)
full-txt-destination Configure destination profile for plain txt message
short-txt-destination Configure destination profile for short txt message
Now we used the pre-defined TAC profile for sending the HTTP requests. As you can see the
configuration is limited of this profile, because it uses a number of defaults.
Next we can finish the configuration of this chapter with configuring an additional
destination-profile for sending e-mails.
SW1
SW1(config-callhome)# destination-profile CiscoTAC-2 transport-method email
SW1(config-callhome)# destination-profile CiscoTAC-2 email-addr
ciscotac@ciscocallhome.com
SW1(config-callhome)# destination-profile CiscoTAC-2 ?
<CR>
alert-group Add alert group
email-addr Add email addr
format Callhome message format (default XML)
http Add http or https url
message-level Callhome message level(0-lowest urgency, 9-highest urgency)
message-size Configure maximum message size (default 2500000)
transport-method Callhome message sending transport-method
Now every destination-profile is correctly configured so the final task is to enable the
callhome configuration.
SW1(config-callhome)# enable
contact phone is not configured
callhome cannot be enabled on the switch,
because necessary configuration has not been done
Please check if all of following configuration is done
contact person name(sysContact)
contact person's email
contact person's phone number
street addr
To configure sysContact, please use snmp-server command
SW1(config-callhome)#
SW1(config-callhome)#
SW1(config-callhome)# phone-contact +111333444555666
SW1(config-callhome)#
SW1(config-callhome)# enable
SW1(config-callhome)#
The enabling of Callhome will warn you when something has been forgotten. In this case we
forgot to configure the phone number. After the phone number has been configured
Now we finished the configuration. Let’s verify if everything is working as expected.
SW1(config-callhome)# show run callhome
version 5.0(3)N1(1a)
callhome
switch-priority 0
email-contact rmur@ipexpert.com
phone-contact +111333444555666
streetaddress Cisco Street 111
destination-profile CRITICAL_MAILS
destination-profile CRITICAL_MAILS format full-txt
destination-profile CiscoTAC-1 transport-method http
destination-profile CiscoTAC-2
destination-profile CiscoTAC-2 format XML
destination-profile CRITICAL_MAILS email-addr callhome@ciscocallhome.com
destination-profile CiscoTAC-1 email-addr ciscotac@ciscocallhome.com
destination-profile CiscoTAC-2 email-addr ciscotac@ciscocallhome.com
transport email smtp-server mail.ciscocallhome.com port 25
transport email from rmur@ipexpert.com
enable
periodic-inventory notification interval 1
SW1(config-callhome)#