Module 4 - Networking

vSphere 4.
0
Module 4 Networking
Emiliano Turra
Product Support Engineering
VMware Confidential

Rev. G
2 vSphere 4- Mod 4 - Slide
Agenda
Module 0 - Product Overview
Module 1 - VI Installation-Upgrade
Module 2 - VirtualCenter
Module 3 - Storage
Module 4 - Networking
Agenda Lessons for Module 4
Lesson 1: vNetwork Distributed Switch
Lesson 2: Private VLAN
Lesson 3: IPv6
Lesson 4: VMXNET Generation 3
Lesson 5: VMDirectPath I/O
Lesson 6: Virtual Machine Communication Interface (VMCI)
Lesson 7: Basic Troubleshooting Tips

vNetwork Distributed Switch
vCenter
vCenter
Standard Switch
Distributed Switch
Distributed Switch Terminology
Terminology (in Red the official names)
DVN - vNetwork
Distributed Virtual Network, is the umbrella name under which the new network
infrastructure components are grouped. The official name that customers will hear
is vNetwork
dvSwitch, DVS or Distributed Virtual Switch - vNetwork Distributed Switch
Abstraction of multiple hosts sharing the same configuration for vSwitches and
portgroups.
vSwitch - vNetwork Standard Switch
The standard virtual switch that is available in ESX 3.x and 4.x without vNetwork
dvPort
Port in a dvSwitch that allows VMs, vnics, VMKernel or Service Console nics.
dvPort status is stored in VC Database, so it is persistent across hosts
dvPortgroup
Collection of DVPorts that share the same configuration.
Distributed Switch
Distributed Switch: this means that the configuration is centralised to vCenter.
All the hosts that belong to a dvSwitch will not need further configuration to
be compliant
Distributed Switch: the behaviour will still be the same (or consistent) with the
vSwitch we are used to deal with:
dvPortgroups, as a set of dvPorts (the dv equivalent of Portgroups as a
set of ports in a vSwitch)
Configuration is inherited from dvSwitch to dvPortgroup (the equivalent of
what happens for vSwitch/Portgroup)
VMs, Service Console interface (vswif) and VMKernel interfaces can be
connected to dvPortgroups as they could be connected to Portgroups in
vSwitches
Hosts still own 2 configuration contexts, which are therefore not administered
centrally via vNetwork:
Service Console and VMKernel interfaces
Physical NICs and their assignment to dvSwitch Uplink groups
Distributed Virtual Switch Architecture
vCenter
ESX 4 ESX 4 ESX 4
Distributed vSwitch
vSwitch vSwitch vSwitch
Distributed vSwitch
vSwitch
Control Plane
Data Plane
Control Plane (CP) and Data Plane, or I/O Plane are separated.
CP, responsible for configuring dvSwitches,dvPortgroups, dvPorts, Uplinks,
NICTeaming and so on, and for coordinating the migration of the ports, runs on
vCenter
DP, responsible for performing the forwarding, runs inside the VMKernel of the ESX
(Default VMware implementation of CP is via hidden vSwitch).
Distributed Virtual Switch Architecture Data Plane
vSwitch
Data Plane
Port Port Port
IO Filter
Forwarding Engine
Teaming Engine
Port Port
IO Filter IO Filter
IO Filter IO Filter
Filters (DVN Switch API, or dvFilter)
Forwarding (DVN Appliance API, or VSafe-net)
Uplink Abstraction
UPLINK groups allow for abstraction from the physical implementation of
each server.
Each Physical host can contribute with up to 1 NIC to each Uplink
group
vCenter will only see the uplink groups when configuring the
Distributed Switch, because each host can contribute in a different
way (vmnic0,1,2,3,)
vCenter
vmnic0,1,2,3,?
Comparing Standard and Distributed Switch
Both
can forward L2 frames
can segment traffic into VLANs
can use and understand 802.1q VLAN
encapsulation
can have more than one uplink (Nic
Teaming)
can have traffic shaping for the
outbound (TX) traffic
Only Distributed Switch
can shape inbound (RX) traffic
has a central unified management
interface through VC
supports Private VLANs (PVLANs)
provides potential customisation of
Data and Control Planes
Standard Distributed
L2 Switch YES YES
VLAN Segmentation YES YES
802.1Q Tagging YES YES
NIC Teaming YES YES
TX Rate Limiting YES YES
RX Rate Limiting No YES
Unified
management interface
No YES
PVLAN No YES
3
rd
Party Virtual Switch Support No YES
Distributed Switch does/does nots
DS is/does
Simplify datacentre setup by centralising network configuration
Will make it easier for VI Admins to add hosts to the cluster and have them
immediately VMotion compatible
Each dvPort is unique across the dvSwitch, and therefore across the
cluster, and will follow the client if it is moved around, for example
VMotion of a VM.
DS is NOT:
A single and whole Standard Switch across hosts, because:
It behaves roughly as if you had Standard Switches configured
consistently across the hosts
The traffic between two VMs on the same dvPortgroup but on different
hosts will still go through the physical network via the Distributed
Switch Uplinks
PVLANs require physical configuration or VMotion will break
connectivity.
Standard Switch + Host Profiles = DS ?
Standard Switch + Host Profiles = Distributed Switch ?
You get all the Standard Switch Features plus the ability
to re-create them on new hosts
No DS features
Manual process of applying new modifications to all the
hosts
There is no Uplink group, so when vmnic names differ
across hosts, configuring nicteaming might be
impossible via one single profile
Changes are applied in maintenance mode
Custom Distributed Switch
vCenter
ESX 4
ESX 4 ESX 4
Distributed vSwitch
vSwitch vSwitch
Control Plane
I/O Plane (Data Plane) and Control Plane can be replaced with 3
rd
party versions
Custom Data Plane implements Forwarding/Filtering/Teaming, basically replacing the
vSwitch
Custom Control Plane is implemented as an appliance, and will be responsible for
handling the configuration of the ports (storing, changing and migrating), and coordinating
the configuration across DPs (across hosts)
Data Plane Agents (DPAs) will run as VMKernel Worlds and will be responsible of
communication between CP and DPs
vC
Extension
vSphere Client
Plugin
Control Plane
Appliance
Data
Plane
DataPlane
Agent
Creating Distributed Virtual Switch - 1
Go to Home > Inventory > Networking
If you are in other locations, the New DVS button is disabled
Create a new Distributed Switch
Specify:
Name of the Distributed Switch
Number of Uplink Ports
Uplinks can be renamed/added afterwards

Creating Distributed Virtual Switch - 2
Add hosts and Uplinks (vmnic groups) from Cluster
An Uplink is to a Distributed Switch what a vmnic is to a Standard Switch
Due to the fact that the Distributed Switch is a logical/abstract entity that
exists across hosts, the association between a Distributed Switch and each
hosts vmnic is done via this further abstraction called Uplink.
What is called Uplink here is a group of vmnics, grouped by the VI
Administrator when adding hosts/vmnics to the Distributed Switch
Creating Distributed Virtual Switch 3
Select whether to create a default Portgroup or not
The Distributed Switch is ready

Uplinks
Assigning Uplinks to a Distributed Switch
Uplinks are associated automatically at Distributed Switch creation time
If changes need to be applied, they have to be applied from the host
Therefore in vCenter, go to Host > Configuration > Networking
Select DVS view
Click on Manage Physical Adapters
If you click on the first <Click to Add NIC>,
the NIC will be added to the Pending Uplink
Assignment group and assigned automatically
when you press Ok
Click on <Click to Add NIC> below the Uplink
group you wish to assign the vmnic to

Managing Distributed Switch
Distributed Switch properties are grouped in 3 tabs:
Properties
General
Advanced
Network Adapters
View Physical adapter contributed by each
member (ESX). No modification allowed from
this screen, you need to go to the specific host
configuration for managing Uplinks
Private VLAN
Where you can associate/edit Primary and
Secondary PVLANs.
Changes might not take place if you try to edit
PVLANs that are in use, disconnect the VMs
first. We will see PVLANs later
Managing Distributed Switch - General
General
Allows you to define (Prompted also at DVS Creation time)
the DVS name,
the number of UPLINK ports,
Additionally, allows you to define
notes
It allows also to edit the Uplink names.
Managing Distributed Switch - Advanced
Advanced
Allows to define:
Max value for Maximum Transmission Unit (Useful for enabling Jumbo Frame)
For the Standard vSwitch, the only options are:
esxcfg-vswitch m and -l
Cisco Discovery Protocol Status
For the Standard vSwitch, the only options are:
esxcfg-vswitch B and -b
Administrators details
Distributed Switch Portgroups
Similarly to what happens with the standard vSwitch, also in a
Distributed Switch Portgroup:
represents a group of Ports that share the same
configuration template.
does not constitute the means to segregate traffic

Settings divided into 3 categories :
General
Policies
Security
Traffic Shaping
VLAN
Teaming and Failover
Miscellaneous
Advanced
Distributed Switch Portgroups - General
General
Allows you to define
The name of the portgroup
A description
The number of ports available
The type of Port Binding, which can be
Static
Dynamic
None (Ephemeral ports)
Port Binding
Static Binding (Default): means that the dvPort will be assigned to
the VM at configuration time. Once all the ports are booked by
VMs, it will not be possible to connect any more VM, independently
from the fact that the connected VMs are powered up or not, and
an error message will be displayed
Dynamic Binding: means that the dvPort will be assigned at the
moment of powering the VM up. This option allows for over
committing the number of dvPorts.
Ephemeral Ports or No Binding: this behaviour has been
introduced to resemble the behaviour in the standard vSwitch. If
you select this option, the number of ports will be automatically set
to 0, and the Portgroup will allocate one port for each connected
VM, up to the maximum number of ports available in the Switch.
Distributed Switch Portgroups Security
Policies (shows all the options below together)
Security
Similar to what we have already seen in the vSwitch, this section allows you to
define security policies for:
Promiscuous mode
Allowing machines to see the traffic of all the other machines in the DVS
Mac address changes
Allows VMs to receive frames with a Mac Address that is different from
the one configured in the VMX
Forged Transmits
Allows VMs to send frames with a Mac Address that is different from the
one specified in the VMX

Distributed Switch Portgroups Traffic Shaping - 1
Traffic Shaping
Allows you to define ingress and egress traffic shaping.
Ingress shaping is a new feature, and available only with DVS
(not on vSwitch)
Distributed Switch Portgroups Traffic Shaping 2
Traffic Shaping concepts:
Average Bandwidth
Target traffic rate cap that the switch will try to enforce. Every time a client
uses less than the defined Average Bandwidth builds up credit.
Peak Bandwidth
Extra bandwidth available, above the Average Bandwidth specified above,
for a short burst. The availability of the burst depends on credit
accumulated so far
Burst Size
Amount of traffic that can be transmitted or received at Peak speed
(Combining Peak Bandwidth and Burst Size you can calculate the
maximum allowed time for the burst)

Distributed Switch Portgroups VLAN - None
VLAN (Allows you to specify the VLAN behaviour of the dvSwitch,
VDS Only):
NONE
Physical equivalent to: No VLAN Tagging
Standard vSwitch equivalent to: VLAN ID option set to 0
EST External Switch Tagging
Distributed Switch Portgroups VLAN Single VLAN
DVS Only):
VLAN
Physical equivalent to: VLAN in Access/Untagged mode
Standard vSwitch equivalent to: VLAN ID option
VLAN ID 4095 is not allowed here
VST Virtual Switch Tagging
Distributed Switch Portgroups VLAN - Trunk
VDS Only):
VLAN Trunking
Physical equivalent to: VLAN in Trunk/Tagged mode
Standard vSwitch equivalent to: VLAN ID set to 4095
VGT VLAN Guest Tagging
VDS Only: option to specify the range of VLANs to trunk, to
improve security.
Distributed Switch Portgroups VLAN - PVLAN
DVS Only):
PVLAN
Physical equivalent to: PVLAN
Standard vSwitch equivalent to: Does not exist
PVLAN
option to specify which Primary and Secondary VLAN to
use (Selecting from the list defined in the Switch)
Distributed Switch Portgroups Teaming & Failover
Teaming and Failover
Allows policies to be defined for:
Load Balancing
Failover detection
Notify Switches
Failback
Failover order
Specific Uplink usage
From the screenshot on the right, you
can see how the Active/Standby status
is applied to each uplink group
(dvUplink1 and 2 in this case), and not
to the vmnics directly, as it used to be
with standard vSwitches
Distributed Switch Portgroups Misc.
Miscellaneous
Allows you to block all the dvPorts of the dvPortgroup, DVS Only
Distributed Switch Portgroups Advanced - 1
The dvPortgroup Advanced subcategory is different from dvSwitch:
It allow each single dvPort to override the settings of the
dvPortgroup. clicking on Edit Override Setting the VI Admin can
also specify which properties to allow/not allow to be overridden at
lower levels.
Distributed Switch Portgroups Advanced - 2
The dvPortgroup Advanced subcategory is different from dvSwitch:
It allow each single dvPort to override the settings of the
dvPortgroup. clicking on Edit Override Setting the VI Admin can
also specify which properties to allow/not allow to be overridden at
lower levels.
Configuring Distributed Switch Virtual Adapters -1
Two types of Virtual Adapters:
Service console vswif
VMKernel vmknic
To use Virtual Adapters inside a dvSwitch, you need to configure them via Host >
Configuration > Networking, as this is not a cluster-wide option.
Select Distributed Virtual Switch view and click on Manage Virtual
Adapters
Youll be prompted with the Manage Virtual Adapters dialog, where you can:
Add a new adapter
If you already have DVS virtual Adapters, youll be able to:
Edit the adapter (IP address/netmask, default gateway, DNS servers)
Migrate it back to a vSwitch
Delete it (Deleting the last vswif is not allowed)
If you click on Add, for each Virtual Adapter type, there will be 2
options:
Create a new Adapter
Migrate the existing from vSwitch to dvSwitch
Either way, youll be prompted to specify an existing dvPortgroup to be
connected to
Migrating from Standard Switches
If after selecting Add, you chose to Migrate existing virtual network adapters, youll
be prompted with the form below
Select which adapters you wish to migrate
For each selected adapter, specify which dvPortgroup you want to connect it to.
The migration will take care of not interrupting the traffic, so for example vCenter wont
show the ESX as disconnected even if you migrate its only vswif interface
Migrating from vSwitches - logs
Example: migrating vswif2 with IP address 192.168.9.1 (Hex 0x109a8c0) from vSwitch0 to
dvSwitch:
cpu1:4175)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.connectid on port 97
cpu3:4177)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.portgroupid on port 97
cpu4:4179)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.block on port 97
cpu4:4170)DVSDev: DVSDevDataSet: clearing data com.vmware.common.port.shaper.input on port 97
cpu4:4168)DVSDev: DVSDevDataSet: clearing data com.vmware.common.port.shaper.output on port 97
cpu1:4167)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.teaming on port 97
cpu3:4178)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.security on port 97
cpu3:4169)DVSDev: DVSDevDataSet: setting data com.vmware.etherswitch.port.vlan on port 97
cpu1:4175)DVSDev: DVSDevDataSet: clearing data com.vmware.etherswitch.port.ipfix on port 97
cpu3:4177)DVSDev: DVSDevDataSet: setting data com.vmware.common.port.statistics on port 97
cpu0:4096)Tcpip_Socket: vmk_set_ip_address:968: index = 145660792, ip_addr = 0x109a8c0, netmask = 0xffffff
cpu1:4109)Mirror: Mirror_PortDisable: removing wildcard INPUT match port vswif2(0x8) from session
legacy_promiscuous
cpu0:4096)Net: NetDisconnect:1250: disconnected from net vSwitch0, PortID = 0x8
Preparing dvPort 97 to receive vswif2

Migrating from vSwitches - logs
cpu0:4096)NetDVS: DVS_PortAssociate:413: Connecting to DVS 49 89 34 50 eb b6 a0 ae-d9 d3 3e
e1 68 b4 5d 45 port 97
cpu0:4096)NetDVS: DVSPortAssociate:1155: port 0x410004256ce0 (type 1)
cpu0:4096)NetDVS: DVS_PortAssociate:438: Connected to DVS port 97 (type 1), dvs 49 89 34 50
eb b6 a0 ae-d9 d3 3e e1 68 b4 5d 45
cpu0:4096)NetPortset: Portset_ConnectPort:1251: newID 0x300000c, newIDIdx 0xc, psMask
0x1ff, newPort 0x41000412db80, portsInUse 6, portCfgName <none>
cpu0:4096)Net: NetConnectCommon:1054: connected to net (null), portset 0x410004004428,
PortID = 0x300000c, status 0x0
cpu6:4111)Net: COSVMKDev_Enable:1419: port = 0x300000c, cosStateVA = 0x41007cb88000,
cosStateVP = 0x41007cb88000, cosStateLen=0x649c
cpu6:4111)Net: COSVMKDev_Enable:1444: txRing = 0x41007cb8949c, rxRing = 0x41007cb8809c,
numRxBufs = 0x80, numTxBufs = 0x80
cpu6:4111)Net: COSVMKDev_Enable:1468: COS VMK gen count = 11
cpu6:4111)Net: COSVMKDev_Enable:1481: Enabling NIC in the shadow vmkernel tcpip stack
cpu6:4111)Tcpip_Interface: vmk_nic_attach:893: ether attach complete
cpu6:4111)NetDVS: DVS_PortLinkUp:501: DVS_PortLinkUp portID 0x300000c DVS port 97
cpu6:4111)NetPort: PortBlockSet:2040: resuming traffic on DV port 97
cpu6:4111)VLAN: VLAN_UpdateDVSPortCfg: VLAN 64 configured for DVPort 50331660
cpu6:4111)etherswitch: NCP_AddBeaconVID: 64
cpu6:4111)Mirror: MirrorSessionWildcardAddPort: adding wildcard match port
vswif2(0x300000c) for INPUT to session legacy_promiscuous
Migrating VMs Between dvPortgroups
VI4 introduces a new feature that allows you to mass-move VMs from
one dvPortgroup to another
To initiate a Migration, go to the Summary page of the dvSwitch (from
Host > Inventory > Networking)
Click on Migrate Virtual Machine Networking
Select Source and Destination dvPortgroup
Click on Show Virtual Machines
Select the VMs you want to Migrate

Migrating to DS Step by Step
vswif0
vmk0
vm1
vm2
vm2
vm2
vm2
vm2
vSwitch0
vSwitch1
d
v
P
G
0

d
v
P
G
1

DS
Uplink1
Uplink2 1
Uplink3
Uplink4
Uplink1
Uplink2
Uplink3
Uplink4
0
Steps:
1. Create a DS with as many Uplink groups as
Physical NICs connected to the Standard
Switches
2. Create in the DS as many Portgroups as
you already have in the SS
3. Assign Uplinks to each Portgroup in the DS
4. Break each teaming and transfer one NIC
from each vSwitch to a corresponding
Uplink group
5. Migrate the Virtual Adapters and the Virtual
Machines to the appropriate Portgroups
6. Transfer the remaining uplinks to the Uplink
groups associated with the appropriate
Portgroups
7. Remove the Standard Switches and their
Portgroups
2
3
Lab Exercise
Lab 1: vNetwork Distributed Switch
Lesson 1: vNetwork (Distributed Virtual Networks)
Lesson 3: IPv6
What are Private VLANs ?
What is a Private VLAN?
VLAN is a mechanism to divide a broadcast domain into several logical broadcast
domains
Private VLAN is an extension to the VLAN standard, already available in several
(most recent) physical switches. What it does is add a further segmentation of the
logical broadcast domain, to create Private groups
Furthermore, because it divides a VLAN (which will be called Primary
PVLAN) into one or more groups (called Secondary PVLANs), this means
that all the Secondary PVLANs exist only within the Primary VLAN.
Private because, depending upon the type of the groups involved, hosts will
not be able to communicate each other, even if they belong to the same group.
Each Secondary PVLAN has an associated VLAN ID, and the physical switch
will associate the behaviour (Isolated, Community or Promiscuous) depending
on the VLAN ID found in each packet.
Secondary Private VLAN Types
Primary Secondary Type
5 Promiscuous
5 155 Isolated
5 17 Community
Host 1
Host 2
Host 3
Host 4
Host 5
Host 6
155
5
17
Three types of Secondary PVLANs:
Promiscuous
A node attached to a port in a
promiscuous secondary PVLAN
may send and receive packets to
any node in any others secondary
VLAN associated to the same
primary. Routers are typically
attached to promiscuous ports.
Isolated
A node attached to a port in an
isolated secondary PVLAN may
only send to and receive packets
from the promiscuous PVLAN.
Community
A node attached to a port in a
community secondary PVLAN
may send to and receive packets
from other ports in the same
secondary PVLAN, as well as
send to and receive packets from
the promiscuous PVLAN.
Private VLAN Implementation
Standard 802.1Q Tagging
No Double Encapsulation
Switch software decides which ports to forward the
frame, based on the tag and the PVLAN tables
5 5 Promiscuous
5 155 Isolated
5 17 Community
VLAN 5
PVLAN 5
(Promiscuous
)
PVLAN
155
(Isolated)
PVLAN 17
(Community)
Why Private VLANs ? Problem
Why PVLANs? (examples)
Machines can be violated/infected, and can be used as a bridge for
violating/infecting other machines in the same network segment
Attacks like ARP poisoning are still a danger, and port-security type of
defence does not work well with ESX (For example in case of VMotion, if
you set port-security to allow a maximum of X different MAC addresses,
when you VMotion a VM that happens to be the X+1th, youll lose
connectivity)
Segmentation of each and every host in the network is required
Internet
Infected Machine,
acting as a bridge
to infect others

Machine that would not
be reachable from Internet
Gateway/Serer
Rogue machine
performing ARP Poisoning
impersonating the gateway
Victim sends
traffic to the
rogue instead
of the gateway
Why Private VLANs ? Solutions
Solutions:
One VLAN per host or group of hosts
CONS:
A a lot of subnets of the /30 type, with waste of IP addresses (50%)
Consequently, lot of routes, which are difficult to maintain and change
Complex and expensive gateway (firewall) rules
Available VLANs are 4095*, but switches allow much less, about 1000
Too complex/expensive to maintain
One VLAN per VM, with one VM acting as transparent/software bridge with
firewall, thus on the same subnet
Can be implemented inside ESX 3.x
Even more complexity/cost
PVLAN
Private VLANs: Example without PVLANs
Internet
Gateway
ISP
For bigger Customers, /29 or /28 Subnets
Several /30 Subnets
Example: Hosting company:
Many different customers that should not be able to see each
other
Possible solution:
One VLAN per customer, but:
Creating a VLAN for each customer is expensive:
One subnet per customer is required, gateway
maintenance is a nightmare
If a customer grows in size, subnets might have
to be changed (for example /30 to /29)
Physical switches can handle a limited amount
of VLANs per switch (less than 4000)
Private VLANs: Example with PVLANs
PVLANs
Single Subnet
Gateway in the promisc PVLAN
Each Customer in Isolated PVLAN
Community PVLAN if Customer expands
Internet
Gateway
ISP
Community for big customer
Isolated for small customers
Promisc for the gateway
Private VLANs & vNetwork - 1
vSphere 4 supports PVLANs if you are using vNetwork (DS)
PVLAN in dvSwitch works like PVLAN in Physical Switches:
Primary VLAN is associated with one or more secondary VLANs
Secondary PVLANs have an additional attribute, which is one of the 3:
Promiscuous
All the machines connected to a Promiscuous PVLAN portgroup will be able
to send to and receive from any other portgroup that is an Isolated or
Community PVLAN associated to the same Primary VLAN
Community
All the machines connected to a Community PVLAN Portgroup can send to
and receive from any other machine on the same Community or
Promiscuos PVLAN associated with the same primary VLAN
Isolated
Each machine connected to an Isolated PVLAN Portgroup can send to or
receive from only machines on the Promiscuous PVLAN associated to the
same primary VLAN
Private VLANs & vNetwork - 2
Promiscuous PVLANs will have the same VLAN ID both for Primary and
Secondary VLAN
Community and Isolated PVLANs traffic will travel tagged as the associated
Secondary PVLAN
Traffic inside PVLANs will not be encapsulated (NO Secondary PVLAN
encapsulated inside a Primary PVLAN Packet)
Traffic between VMs on the same PVLAN but on different ESX will go through
the Physical Switch
Therefore the Physical Switch must be PVLAN aware and configured
appropriately, in order to allow the secondary PVLANs to reach destination.
5 5 Promiscuous
5 155 Isolated
5 17 Community
PVLAN and Physical Switch
Because of the PVLAN implementation, packets travel tagged with the
secondary ID, and each VM can receive and send to different secondary
PVLANs (For example Community and Promiscuous)
Physical Switch can be confused by the fact that each mac address is
visible in more than 1 VLAN tag
Physical switch is REQUIRED to be PVLAN aware, and to have the
same PVLAN mapping as the vDS
Still, the physical switch must trunk to the ESX, and NOT be in a
secondary PVLAN!
PVLAN in the vDS will work even with non PVLAN aware physical
switches if these are not discovering mac addresses per VLAN
Because this way the mac address is associated to the single port.
PVLAN and Physical Switch - Example
Example: a VM in a Promiscuous
PVLAN tries to do an ARP request
for a VM in an Isolated PVLAN, on
a different ESX, and the Physical
Switch is not PVLAN aware.
dvSwitch
Isolated
Promisc
Arp request
Tag: none
5 5 Promisc
5 155 Isolated
5 17 Comm
Arp request
Tag: 5
PVLAN logic detects that the
destination is Isolated
so act as if the tag were 155
Arp request
Tag: none
Arp Reply
Tag: none
Arp Reply
Tag: 155
Arp Reply
Tag: 155
Switch ports that see
the same mac address
through different VLAN tags
Arp Reply
Tag: none
Arp request
Tag: 5
Private VLANs Isolated
5 5 Promisc
5 155 Isolated
5 17 Comm
VM 1
VM 5
VM 4
155
5
17
155
5
5
Physical
dvSwitch
VM 3
VM 2
VM 6
VM 1 cant talk to any VM
in PVLAN 155
in PVLAN 17
VM 1 can talk to VMs
in PVLAN 5 in
Virtual Switches
Physical Switch
VM 1 can talk to VM 2 and 3
only if the physical switch is
configured to handle PVLAN
155.
If the Physical switch allows
VLAN 155, the isolation might
be compromised.
Private VLANs Community
5 5 Promisc
5 155 Isolated
5 17 Comm
VM 7
VM 5
VM 4
155
5
17
5
5
Physical
dvSwitch
VM 3
VM 2
VM 6
VM 7 cant talk to any VM
in PVLAN 155
VM 7 can talk to VMs
in PVLAN 17
in PVLAN 5 in
Virtual Switches
Physical Switch
VM 7 can talk to VM 2 and 3
only if the physical switch is
configured to handle PVLAN
17.
If the Physical switch allows
VLAN 17, the isolation might
be compromised.
17
Creating Private VLANs
Create the PVLAN table in the dvSwitch
Edit Properties fo the dvSwitch, and select the PVLAN Tab
On the Primary Tab, add the VLAN that will be used outside the PVLAN
domain, and select it
On the Secondary Tab, create the PVLANs of the desired type. There
can be only one Promiscuous PVLAN and is created automatically for
you.
Beware: before deleting any primary/secondary PVLAN, make sure
that they are not in use, or the operation will not be performed.
Lab Exercise
Lab 2: Using PVLANs

Break
Lesson 3: IPv6
IPv6
IPv6 Concepts
VI4 and IPv6
New TCP/IP Stack
GuestOS and IPv6
IPv6 Concepts - 1
IP Next Generation (v4 was officialised in 1981)
Addresses are 128-bits long
Example: localhost (127.0.0.1) now is:
0000:0000:0000:0000:0000:0000:0000:0001
or ::1 for short (:: means pad with zeros)
fe8x: fe9x: feax: febx: are Link-local addresses (will never be
routed), similar to RFC 3927 defined 169.254/16 range
fecx: fedx: feex: fefx: are Site-Local addresses (similar to private
IPs in IPv4, such as 10.0.0.0/8). The Site-Local addresses are
deprecated by RFC 3879 in production but still valid for labs, for
example
IPv6 Concepts - 2
No more IP broadcasts, but advanced multicast
IPv6 has autoconf capabilities, and via multicast can discover routers
and receive the configuration from them.
There is also an IPv6 version of DHCP.
DNS can serve IPv6 entries, even over IPv4 connections (or vice
versa).
IPv6 can be tunnelled over IPv4, but they cant be mixed (you cant
access an IPv6 host via an IPv4 network, only across an IPv4 network
via tunnels.
IPv6 Concepts: DNS and IPv6

DNS records can be IPv4 (A) or IPv6 (AAAA)
$ dig www.ipv6.org AAAA

; <<>> DiG 9.5.0-P2 <<>> www.ipv6.org AAAA
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57681
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;www.ipv6.org. IN AAAA
;; ANSWER SECTION:
www.ipv6.org. 3600 IN CNAME shake.stacken.kth.se.
shake.stacken.kth.se. 3600 IN AAAA 2001:6b0:1:ea:202:a5ff:fecd:13a6

;; AUTHORITY SECTION:
stacken.kth.se. 3600 IN NS primary.se.
stacken.kth.se. 3600 IN NS secondary.se.
stacken.kth.se. 3600 IN NS b.ns.kth.se.
stacken.kth.se. 3600 IN NS ns.stacken.kth.se.

;; Query time: 671 msec
;; SERVER: 10.21.64.212#53(10.21.64.212)
;; WHEN: Tue Nov 4 16:21:06 2008
;; MSG SIZE rcvd: 174
VI4 and IPv6 - 1
ESX 3.5 added support for IPv6 for VMs
NO TSO (TCP Segmentation Offload) with IPv6
VI 4 adds full VI IPv6 support:
Service Console
VMWare Tools (to display the ipv6 address in vCenter)
VMKernel (and therefore VMotion)
IPv6 Storage (software iSCSI and NFS) is experimental
vCenter will display correctly IPv6 addresses for Service Console, VMKernel and
VMs as reported by the tools
VI4 and IPv6 - 2
What is still not supported in IPv6 in VI4
VI CLI (previously known as RCLI). Configuring IPv6 parameters works, connecting
does not.
CIM
Disabled by default,
Enable via GUI: Host > Configuration > Networking > Properties
Enable for VMKernel (also in VI CLI)
esxcfg-vmknic -6 true
Enable for Service Console
esxcfg-vswif -6 true
Enabling IPv6 on the ESX does not disable IPv4
VI4 and IPv6 3
To edit IPv6 addresses assigned to Service Console or VMKernel adapters,
Go under Host > Configuration > Networking
Select Virtual Switch or Distributed Virtual Switch as appropriate
Edit the vswif interface
IPv6 Address Dialog box:
The box where you can enter the
IPv6 address is free-form.
There is no more the concept of
subnet mask, but subnet prefix,
which is the number of bits that
constitute the prefix (Similar to
CIDR notation for IPv4)
Verifying IPv6 Activation ESX Classic
New VMKernel module: tcpip2

IPv4 module is loaded by default
Based on FreeBSD 6.1
Improved performance and scalability due to locking and threading
improvements (more CPUs can be used)
If IPv6 is enabled for the VMKernel, it will look like this:

For the Service Console, lsmod will contain ipv6 if enabled:

# vmkload_mod -l
Name R/O Addr Length R/W Addr Length ID Loaded
tcpip2v6 0x4180225fd000 0xbd000 0x417fe3676f80 0x37000 47 Yes
# vmkload_mod -l
Name R/O Addr Length R/W Addr Length ID Loaded
tcpip2 0x4180157ed000 0x63000 0x417fd687ac80 0x26000 46 Yes
# lsmod
Module Size Used by
ipv6 259232 18
Note: esxcfg-module -l is equivalent to
vmkload_mod -l, and is available also in the vi-cli.
Verifying IPv6 Activation - ESXi
With ESXi you have two possible ways for checking IPv6 activation:
By logging into the ESX itself, either via the unsupported mode,
or the unsupported ssh connection, and using the same command
as per the ESX Classic:
vmkload_mod -l
By using the vi-cli (also available in the vMA), with the command:
esxcfg-module l

Since there is no service console here, the lsmod part is not
necessary.
$ esxcfg-module -l --server esxi.vmware.com --username root --password secret
Name ID Loaded
tcpip2 45 Yes
$ esxcfg-module -l --server esxi.vmware.com --username root --password secret
Name ID Loaded
tcpip2v6 45 Yes
GuestOS and IPv6
IPv6 support does not require just OS support, applications need to be
made compatible as well!
In Linux, IPv6 is supported since 2.4 but the implementation is not fully
compliant until 2.6 versions
In Windows,
2003 SP1 and XP SP2 have the infrastructure for IPv6, even
though some components of the system and applications are not
IPv6-ready. (For 2003 check http://technet.microsoft.com/en-
us/library/cc776103.aspx)
Vista and 2008 fully support IPv6

Windows and IPv6 addresses: ipv6-literal.net
Most versions of Internet and Windows Explorer do not support literal
IPv6 addresses as described in RFC 2732 (because the colon : is a
reserved character), so DNS AAAA records must be used (for example
for IPv6 web-access to the ESX).
Microsoft has registered ipv6-literal.net as a workaround. The builtin
resolver in windows will intercept this domain and resolve it
automatically, giving access to the corresponding IPv6 address. For
example, the ip address 2001:db8:28:3:f98a:5b31:67b7:67ef would be
accessible as
2001-db8-28-3-f98a-5b31-67b7-67ef.ipv6-literal.net
GuestOS and IPv6 Linux -1
Make sure IPv6 is enabled by checking whether the ipv6 module is loaded using the lsmod
command. If it is not, you might have it disabled in /etc/modprobe.conf, with a line such as:
alias net-pf-10 off that should be removed (A reboot is required)
In RedHat based distributions, including the Service Console:
/etc/sysconfig/network contains the general information regarding network, including
default gateway:
NETWORKING=yes
HOSTNAME=phobos.vmware.com
GATEWAY=10.21.67.254
GATEWAYDEV=eth0
IPV6_AUTOCONF=no
NETWORKING_IPV6=yes
IPV6_DEFAULTGW=fec0::1
IPV6_DEFAULTDEV=eth0
/etc/sysconfig/network-scripts/ifcfg-eth0 contains the information to configure
both IPv4 and IPv6, for example:
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
BROADCAST=172.16.5.255
NETMASK=255.255.255.0
DHCPV6C=no
IPADDR=172.16.5.99
IPV6ADDR=fec0::d/112
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6_DEFAULTGW can have a %eth0 appended at the
end, thus overriding IPV6_DEFAULTDEV
IPV6_AUTOCONF specifies whether IPV6 advertising
should be used to configure NICs
IPV6_ADDR contains also the prefix size (similar to
IPv4 Netmask, in CIDR format)
GuestOS and IPv6 Linux - 2
In Debian based distributions, such as Ubuntu:
The file /etc/network/interfaces contains IPv4 and IPv6 for each interface,
for example:
iface eth0 inet6 static
address fec0::d
netmask 112
gateway fec0::1
IPv6 commands will generally have a -6 option or a 6 at the end to distinguish from the
IPv4 equivalents
ip
ip -6 address add fec0::5/112 dev eth0
ip -6 route add default via fec0::1
ping
ping6 fec0::1
tracepath
tracepath6 fec0::1
traceroute
traceroute6 fec0::1
iptables
ip6tables
GuestOS and IPv6 Windows
In Windows, IPv6 is not enabled by default. You will need netsh to configure it, so we
will see how to enable IPv6 with it as well.
Enable IPv6
netsh interface ipv6 install
Identify the vNIC name, for example in the Network Connections (where you can
also rename it), or with the netsh command
netsh interface show interface
In this example, we will imagine it is Local Area Connection (the default name)
Add an IPv6 address to the selected interface
netsh interface ipv6 add address "Local Area Connection" fec0::1
Add a route for the newly added IP address
netsh interface ipv6 add route fec0::/112 "Local Area Connection
netsh has several dump commands you can use to get information
netsh interface ipv6 dump
netsh interface dump
Lab Exercise
Lab 4: IPv6
Lesson 3: IPv6

VMXNET Generation 3
New state of the art Virtual Network Adapter
Also known as Advanced VMXNET
Based on Enhanced VMXNET introduced in ESX 3.5
Introduces new features:
IEEE 802.1Q VLAN Tagging.
No more need for e1000 in such a case
VLAN Tagging and Tag removal offloading
Only one VLAN per NIC for Windows
TCP Segmentation Offloading for IPv4 and IPv6
TCP and UPD Checksum Offloading for IPv4 and IPv6
MSI (Messaged Signalled Interrupt) and MSI-X support (subject to
guest kernel support)
Receive Side Scaling (supported in Windows Vista, 2008 and any other
system using NDIS 6.x)
VMXNET Generation3
No Record/Replay support
Supported Guest OSes (both 32-bit and 64-bit versions):
All Windows 2003 variants
Windows 2008 variants
Vista and Vista SP1
Windows XP Professional
RHEL 5.x
SLES 10
Ubuntu 7.04+ 8.04, 8.10
Solaris 10 U4 and later
Lab Exercise
Lab 5: VMXNET Generation 3
Lesson 3: IPv6
VMDirectPath I/O - 1
VMDirectPath I/O is a mechanism by which VMs are allowed to directly access
a physical device using the native driver in the GuestOS. Each Device will be
accessible by one single VM.
Main use cases for this feature are I/O devices that may have high
performance/low-latency/CPU efficiency requirements
VMDirectPath I/O (Also known as Fixed Passthrough) is
fully supported for networking I/O devices with the Intel 82598 10
Gigabit Ethernet Controller and Broadcom 57710 10 Gigabit
Ethernet Controller
experimentally supported for storage I/O devices with the QLogic
QLA25xx 8Gb Fibre Channel and the LSI 3442e-R and 3801e (1068
chip based) 3Gb SAS adapters.
VMDirectPath I/O - 2
Support will be limited to Intel and AMD CPUs with EPT/NPT/RVI and IOMMU (VT-d for
Intel) support
The following features are unavailable:
VM cant be VMotion-ed (Uniform Pass Through will allow VMotion, but it is not
available in vSphere 4.0)
Therefore, DRS (limited availability The virtual machine can be part of a cluster, but
cannot migrate across hosts)
Hot add/remove of virtual devices
Suspend and Resume
Record and Replay
Fault Tolerance
High Availability
Memory Overcommitment and Page Sharing
VMDirectPath I/O : Configuring Devices - 1
ESX supports direct PCI device connection for virtual
machines running on Intel Weybridge and Stoakley
platforms. Each virtual machine can connect to up to
two pass-through devices. To configure pass-through
devices on an ESX host:
1. Select an ESX host from the inventory panel of the
VI Client.
2. On the Configuration tab, click Advanced
Settings.
The Pass-through Configuration page appears,
listing all available pass-through devices. A green
icon indicated that a device is enabled and active.
An orange icon indicates that the state of the
device has changed and the host must be
rebooted before the device can be used.
3. Click Edit.
4. Select the devices and click OK.
Once you click Edit, Select the devices you
want to use for VMDirectPath I/O and Click
Ok.
All the dependent devices will be also
configured the same way (wether used by the
VMKernel or used for VMDirectPath).
These devices will be automatically selected
for you.

The configured devices become Orange
You will need to reboot for the devices to become ready (Green)
After the reboot, the devices are green, and ready to be used in a VM
Note: the configuration changes will go into /etc/vmware/esx.conf. In the
case above, the PCI slot where the device was connected is 00:0b:0, so it will be:
/device/000:11.0/owner = "passthru (0b is 11 in decimal)
VMDirectPath I/O : Configuring VM - 1
To configure a PCI device on a virtual machine
Select a virtual machine from the inventory panel of the VI Client.
From the Inventory menu, select Virtual Machine > Edit Settings.
Select the Hardware tab
click Add
Select PCI Device
click Next.
VMDirectPath I/O : Configuring VM - 2
From the list, select the pass-
through device you wish to
assign to the VM.
Once the device is assigned, the
VM must have a memory
reservation for the full configured
memory size.

VMDirectPath I/O : Logs - 1
VMDirectPath I/O requires IOMMU feature in the hosts chipset
Check that the vtd module is loaded, using vmkload_mod l (for ESXi available only
on the console) or esxcfg-module l (Available in VI CLI)
If the module is not loaded, you either do not have the correct/supported chipset, or
there was an issue when loading the module. To find more information on what
happened, you can either attempt to load the module or check the boot logs:
Check /var/log/boot-logs/sysboot.log (/var/log/messages for ESXi)
Locate the sysboot: iommu ... section:
The log example below was taken from a machine using AMDIommu, the experimental AMD
based IOMMU chipset, the module will be vtd at GA time (as with Intel chipsets already):
vmkernel: 0:00:00:51.143 cpu2:4875)ForkExec: UWVMKSyscall: ForkExec:2936:
/sbin/vmkload_mod
vmkernel: 0:00:00:51.178 cpu0:4876)Loading module AMDIommu ...
vmkernel: 0:00:00:51.205 cpu0:4876)AMDIOMMU: ule:428: Loading AMD IOMMU driver...
vmkernel: 0:00:00:51.212 cpu0:4876)AMDIOMMU: ule:438: AMD IOMMU driver version 1.22, built
on: Oct 27 2008
VMDirectPath I/O : Logs - 2
vmkernel logs showing the device being assigned to VMDirectPath I/O:
vmkernel: 0:00:09:16.642 cpu0:7662)PCI: ChangeDevOwnership:1336: 004:00.0 to passthru
vmkernel: 0:00:09:16.649 cpu0:7662)VMK_PCI: vmkpci_PCIDeviceCallback:285: device 004:00.0
event: Device changed ownership: new owner vm
vmkernel: 0:00:09:16.661 cpu0:7662)VMK_PCI: vmk_PCIGetDeviceName:625: Device 004:00.0
name: vmnic0
vmkernel: 0:00:09:16.669 cpu0:7662)LinPCI: LinuxPCIDeviceRemoved: Remove 004:00.0 vmnic0
vmkernel: 0:00:09:16.676 cpu0:7662)WARNING: LinPCI: LinuxPCIDeviceRemoved: no driver (or
not hotplug compatible)
vmkernel: 0:00:09:16.687 cpu0:7662)LinPCI: LinuxPCIDeviceRemoved: Removed device 004:00.0
at event ownership-changed.
VMs vmware.log showing the VM is correctly configured to access the
device:
vmx| DICT pciPassthru0.present = TRUE
vmx| DICT pciPassthru0.deviceId = 1639
vmx| DICT pciPassthru0.vendorId = 14e4
vmx| DICT pciPassthru0.systemId = 4872045d-4d63-ad8e-7fbd-0010182a0a6c
vmx| DICT pciPassthru0.id = 04:00.1
[]
vmx| Registering device pciPassthru0 (A6F3488)

Lab Exercise
Lab 6: VMDirectPath I/O
Lesson 3: IPv6
Virtual Machine Communication Interface - 1
The Virtual Machine Communication Interface (VMCI) is an infrastructure that provides
fast and efficient communication between a virtual machine and the host operating
system and between two or more virtual machines on the same host.
The VMCI SDK facilitates development of applications that use the VMCI infrastructure.
Without VMCI, virtual machines communicate with the host using the network layer.
Using the network layer adds overhead to the communication. With VMCI
communication overhead is minimal and different tasks that require that communication
can be optimized.
Virtual Machine Communication Interface - 2
To enable VMCI on your virtual machine, add the following two lines to the
virtual machine configuration file (.vmx file):
# The following line is REQUIRED.
vmci0.present = "TRUE"
# The following line is OPTIONAL.
vmci0.id = "num"
num is a positive integer that is unique for each virtual machine on your
host. That is, for any virtual machine, you can choose a number (1, 2, 3,
etc.) but two virtual machines must not have the same number as their
vmci0.id.
You also need the VMCI component of the VMware Tools to be installed
inside the VM
VMCI What is it?
Two types of communication
Datagrams
connectionless Similar to UDP
Queue Pairs
Connection oriented Similar to TCP
VMCI provides Socket APIs, which is extremely similar to what is
already used for TCP/UDP applications
IP addresses are replaced with VMCI ID numbers
For example, it has been possible to port netperf to use VMCI sockets
instead of TCP/UDP

VMCI: Use Case
Application server VM connected to a Database server VM.
Internal network can transmit an average of slightly over 2Gbit/s using
vmxnet3
VMCI can go up to nearly 10Gbit/s with 128k sized Queue pairs

Stream Socket Throughput (netperf TCP_STREAM)
0
2
4
6
8
10
12
128 256 512 1024 2048 4096 8192 16384 32768 65536
Message Size
g
b
p
s
VMCI Sockets
VMCI Sockets (128k QP)
TCP/IP (over vmxnet3)
Break
Lesson 3: IPv6
Basic Troubleshooting Tips
VMX Changes
net-dvs, a tool to work with dvSwitch (Beware: not supported)
How to find out about dvPortgroups
esxcfg-vswitch
esxcfg-vswif
esxcfg-vmknic
esxcfg-route
Private VLANs
Cisco Nexus 1000V
esxcfg-firewall
Maximums
Known Issues
VMX changes
DVS
ethernet1.dvs.switchId = "7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1"
ethernet1.dvs.portId = "1423"
ethernet1.dvs.connectionId = "419447540"
ethernet1.dvs.portgroupId = "dvportgroup-302

VMXNET3
ethernet0.virtualDev = "vmxnet3

net-dvs output
switch 7a f2 34 50 21 55 6c 70-a4 b1 10 f1 3f 9d 2c c1 (etherswitch)
Global properties:
com.vmware.common.alias = dvSwitch
com.vmware.common.uplinkPorts =
Uplink1,Uplink2,Uplink3,Uplink4
com.vmware.common.host.uplinkPorts =
5,6,7,8
com.vmware.etherswitch.pvlanMap =
(11, 11) - Promiscuous
(11, 12) - Community
(11, 13) - Isolated
(68, 68) - Promiscuous
(68, 681) - Isolated
(68, 682) - Community
com.vmware.etherswitch.mtu = 0xdc. 5. 0. 0
com.vmware.etherswitch.cdp = 0x 0. 1
com.vmware.common.pgmap =vSwitch-DVUplinks-211:dvportgroup-
212,PVLAN-11-I:dvportgroup-239,PVLAN-11-C:dvportgroup-240,VGT:dvportgroup-
241,PVLAN-11-P:dvportgroup-242,VLAN68:dvportgroup-243,PVLAN-68-I:dvportgroup-
244,PVLAN-86-C:dvportgroup-245,PVLAN-68-P:dvportgroup
-246,Ghost:dvportgroup-299,dvPortGroup:dvportgroup-300,VLAN64:dvportgroup-302
Host properties:
com.vmware.common.host.portset = DvsPortset-1

dvSwitch
identifier
Uplink Identifiers
DVPorts used for Uplink
PVLAN map
MTU
1500 = 0x5DC
(beware of
endian-ness)
CDP Enabled 0/1
dvPortgroup Map, associating
vCenter dvPortgroup names
and dvPortgroup labesl
dvSwitch
Name
net-dvs output
port 5
com.vmware.common.port.alias = Uplink1
com.vmware.common.port.connectid = 1912494964
com.vmware.common.port.portgroupid = dvportgroup-212
com.vmware.common.port.block = false
com.vmware.etherswitch.port.teaming =
load balance = source virtual port id
link selection: link state up; link speed>=10Mbps;
link behavior: notify switch; reverse filter; best effort on failure; shotgun on failure;
active:
standby:
com.vmware.etherswitch.port.security = 0x 1. 0. 0. 0
com.vmware.etherswitch.port.vlan = Guest VLAN tagging
ranges: 1-4094
com.vmware.common.port.statistics:
pktsInUnicast = 1699111
bytesInUnicast = 865718684
pktsInMulticast = 2204789
bytesInMulticast = 580474616
pktsInBroadcast = 7441346
bytesInBroadcast = 623725320
pktsOutUnicast = 1091384
bytesOutUnicast = 783242007
pktsOutMulticast = 34
bytesOutMulticast = 2744
pktsOutBroadcast = 2069749
bytesOutBroadcast = 179071956
pktsInDropped = 159
pktsOutDropped = 0
pktsInException = 1285
pktsOutException = 0
com.vmware.common.port.volatile.vlan = VLAN 0
ranges: 1-4094
com.vmware.common.port.volatile.status:inUse linkUp portID = 0x2000002
net-dvs output
port 519
com.vmware.common.port.alias =
com.vmware.common.port.connectid = 1502730467
com.vmware.common.port.portgroupid = dvportgroup-241
com.vmware.common.port.block = false
com.vmware.etherswitch.port.teaming =
load balance = source virtual port id
link selection: link state up; link speed>=10Mbps;
link behavior: notify switch; reverse filter; best effort on failure; shotgun on failure;
active: Uplink1 Uplink2 Uplink3 Uplink4
standby:
com.vmware.etherswitch.port.security = 0x 0. 0. 0. 0
com.vmware.etherswitch.port.vlan = Guest VLAN tagging
ranges: 11-14 64-72
com.vmware.common.port.volatile.persist = /vmfs/volumes/f1c540c6-3bd757e8/.dvsData/7a f2 34 50 21 55 6c 70-a4
b1 10 f1 3f 9d 2c c1/519
com.vmware.common.port.volatile.vlan = VLAN 0
ranges: 11-14 64-72
com.vmware.common.port.statistics:
pktsInUnicast = 3972
bytesInUnicast = 571094
pktsInMulticast = 27
bytesInMulticast = 2166
pktsInBroadcast = 17
bytesInBroadcast = 2712
pktsOutUnicast = 6499
bytesOutUnicast = 7405784
pktsOutMulticast = 2488
bytesOutMulticast = 664816
pktsOutBroadcast = 1103380
bytesOutBroadcast = 95151238
pktsInDropped = 0
pktsOutDropped = 0
pktsInException = 503
pktsOutException = 0
com.vmware.common.port.volatile.status:inUse linkUp portID = 0x200000d

net-dvs notes
Launch with /usr/lib/vmware/bin/net-dvs
Output collected by vm-support
Not Available for ESXi unless you connect directly via SSH (Not
supported)
DVS information is cached in /etc/vmware/dvsdata.db
Binary file
Collected by vm-support
Can be used to produce net-dvs output from any linux host (for example
scripts server) with the net-dvs f [FILE] command
DVS Port information is stored in a shared VMFS volume root, under
.dvsData/, net-dvs output will indicate the exact location. This can be useful to
quickly locate which ports are still accessing a given DSwitch
References to the DVS are also on /etc/vmware/esx.conf
VMKernel ports
DVS Information in vCenters DB
DvPortgroups are defined at vCenter level, there is no way to gather
information about them from the host.
In vCenters database, you can find out about dvPortgroups with:
select * from VPX_DVPORTGROUP
Do not alter the contents of the table in any way! If you remove anything,
you might not be able to clean up the ghost ports anymore.

esxcfg-vswitch
#esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 32 2 32 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 3 64 1500 vmnic1

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 1 vmnic1

DVS Name Num Ports Used Ports Configured Ports Uplinks
dvSwitch 64 6 512 vmnic3,vmnic2

DVPort ID In Use Client
5 1 vmnic2
6 1 vmnic3
7 0
8 0
391 0
390 0
1422 1 vmk0
1419 1 vswif1
1423 1
519 0
1420 0

Applies also for ESXi via VI CLI
esxcfg-vswif
Create a new vswif
Same syntax as ESX 3.x
esxcfg-vswif -a vswif1 -i 10.21.64.25 -n 255.255.252.0 -p Service Console
For DVS youll need to specify dvSwitch name and dvPort:
esxcfg-vswif -a vswif0 -i 10.21.64.125 -n 255.255.252.0 -P 1421 -V dvSwitch
IPv6 (supposing IPv4 already configured)
esxcfg-vswif -i fec0::4/112 vswif1
IPv6 with DHCP (supposing IPv4 already configured)
esxcfg-vswif -i DHCP6 vswif1

Output of esxcfg-vswif l
Name Port Group/DVPort IP Family IP Address Netmask Broadcast Enabled TYPE
vswif1 1419 IPv4 10.21.64.25 255.255.252.0 10.21.67.255 true STATIC
vswif1 1419 IPv6 fec0::4 112 true STATIC
vswif1 1419 IPv6 fe80::250:56ff:fe4f:cba 64 true STATIC
Does not apply for ESXi via VI CLI
esxcfg-vmknic
Add a vmknic on a vSwitch
esxcfg-vmknic a -i 10.21.66.25 -n 255.255.252.0 p VMKernel Network
Add a vmknic on a DVS (dvPort 1422)
esxcfg-vmknic a -i 10.21.66.25 -n 255.255.252.0 -s dvSwitch -v 1422
Add an IPv6 address to the newly created vmknic
esxcfg-vmknic -i fec0::5/112 -s dvSwitch -v 1422
Add an IPv6 DHCP address to the newly created vmknic
esxcfg-vmknic -i DHCP6 -s dvSwitch -v 1422
Output of esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask
Broadcast MAC Address MTU TSO MSS Enabled Type
vmk1 1421 IPv4 10.21.66.25
255.255.252.0 10.21.67.255 00:50:56:75:79:ae 1500 65536 true STATIC
vmk1 1421 IPv6 fe80::250:56ff:fe75:79ae 64
00:50:56:75:79:ae 1500 65536 true STATIC
vmk1 1421 IPv6 fec0::5 112
00:50:56:75:79:ae 1500 65536 true STATIC

esxcfg-route
Add an IPv6 default gateway (all the other operations are the same as 3.5)
esxcfg-route -f V6 -a default fec0::1
Display IPv6 routes for VMKernel
esxcfg-route -f V6 -l

VMkernel Routes:
Network Netmask Gateway
default 0 fec0::1
fe80:: 64 Local Subnet
fec0:: 112 Local Subnet
ff01:: 32 Local Subnet
ff02:: 32 Local Subnet

Troubleshooting PVLANs
Key concepts to keep in mind when troubleshooting PVLANs:
Packets in PVLANs travel tagged as if they were in a VLAN with ID as
the Secondary ID, there is no encapsulation. This is valid for both virtual
and physical switches
Physical switches need to be configured to forward packets in such VLAN
IDs between source and destination
Consider PVLAN as a particular case of VST, so:
Physical switch to ESX should be trunking
Physical switches should be connected via trunks
Unless they are not PVLAN aware, in which case the trunk
should be a PVLAN trunk if you are using Isolated PVLANs
Physical hosts should be connected to a PVLAN port
VTP (Vlan Trunking Protocol) has to be in transparent mode in the
physical switch, because PVLANs are defined locally on the single
physical switch
Troubleshooting PVLANs
Troubleshooting hints
Make sure that the physical and virtual switch configuration matches:
Physical switch port is trunking for all the primary and secondary
PVLAN IDs
Compare the PVLAN maps in physical and virtual switch
In Cisco switches, you can use the commands:
show running-configuration
show interface private-vlan mapping
show interface [interface-id] switchport

PVLAN in Physical Switches
CISCO IOS
Create the primary PVLAN (in this example VLAN 11)
(config)# vlan 11
(vlan-config)# private-vlan primary
Similarly, create the secondary PVLAN (ex. VLAN 13, Isolated, 12, Community)
(config)# vlan 13
(vlan-config)# private-vlan isolated
(config)# vlan 12
(vlan-config)# private-vlan community
Bind Primary and Secondary PVLANs
(config)# vlan 11
(vlan-config)# private-vlan association 12,13
Bind switch ports to the PVLANs (1/10 Isolated, 1/11 Community and 1/1 promisc):
(config)# interface Fastethernet 1/10
(config-if)# switchport mode private-vlan host
(config-if)# switchport private-vlan host-association 11 12
(config-if)# switchport mode private-vlan host
(config-if)# switchport private-vlan host-association 11 13
(config-if)# switchport mode private-vlan promiscuous
(config-if)# switchport private-vlan mapping 11 12,13
Cisco Nexus 1000 SVSTroubleshooting
If traffic doesnt work, try the following:
On the CP, check that the DP module is visible.
# show module (Should show the UUID of the ESX 4.0
host.)
# show server_info (Should show the hostname of the ESX 4.0
host).
Ensure that your uplinkportprofile1 includes the VLAN that is
configured on your VMs port profile.
# show port-profile name uplinkportprofile1
To isolate how far the traffic gets, do tcpdump inside the VMs, cb
print ingress on the DP, and debug ip packets detail on the
upstream Cisco switch
esxcfg-firewall - 1
New feature (soon available also in 3.5) of filtering connections per host/port,
with the option:
--ipruleAdd <host,cport,tcp|udp,REJECT|DROP|ACCEPT,name>

As you might already know from ESX 3.x, list firewall rules with esxcfg-
firewall q and be careful, because -l will reload the firewall rules instead,
overwriting the possible root cause of your investigation.

There is no mechanism to temporarily stop the firewall like in ESX 3.5 using
service firewall stop|start because the service firewall stop
will not do anything but print the following:
firewall can't be stopped. To disable the firewall run, esxcfg-firewall
--allowIncoming allowOutgoing

esxcfg-firewall - 2
But keep in mind that:
esxcfg-firewall --allowIncoming allowOutgoing modifies the
firewall configuration, so to return to the previous configuration you need
to use esxcfg-firewall --blockIncoming --blockOutgoing,
because esxcfg-firewall -l wont.
If you use allowIncoming and allowOutgoing, previously defined IP Rules
will still be applied
esxcfg-firewall - 3
So what can we do for temporarily disabling the firewall for troubleshooting?
Remember to save the actual configuration before doing anything else!
Otherwise you might not be able to identify the root cause.
Save the output of iptables -L or better of iptables-save to a file.
You can use iptables -F or iptables-save, and then reload the
firewall with esxcfg-firewall l, when the troubleshooting is done.
esxcfg-firewall - 3
With iptables F youll flush all the rules. Keep in mind that usually the
default policy is to drop connections, and the rules are allowing you in. This
means that before flushing the rules, you should make sure that at least the
INPUT chain has default set to ALLOW, with iptables P INPUT ALLOW, or
youll lock yourself out.
With iptables-save>file you can save to a file the rules, then edit the files
so that you remove all the rules and the chains, edit the policy to be ALLOW,
review what youve done, and then apply your changes with iptables-
restore<file. For example:
*filter
:INPUT ACCEPT [4495370:1545008248]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3029364:951838897]
COMMIT
Maximums - 1
Maximums
VI3
Standard
Switch
vNetwork
Standard
Switch
vNetwork
Distributed
Switch
Switches per VC 4096 4096 16
Switches per ESX host 248 248 16
Port groups per ESX host 512 512 512
Port groups per switch 512 512 512
Ports per host 4096 4096 4096
Uplinks per host 32 32 32
Ports per switch 1016 1016 8000
Uplinks per virtual switch 32 32 32
Max number of hosts per switch NA NA 300
VLANs/Private VLANs Limited by Max # of Portgroups
Maximums - 2
Physical NIC Type
Max Number
of ports per
ESX Host
tg3 (Broadcom 1GigE) 32
bnx2 (Broadcom 1GigE) 16
e1000e (Intel 1GigE PCIe) 32*
s2io (Neterion 10GigE) 4
e1000 (Intel PCIx) 32*
nx_nic (Netxen 10GigE) 4
Igb (Intel Zoar) 16
bnx2x (10GigE Broadcom) 4
igbe (Intel 10GigE Oplin) 4
(*) If the Hardware supports them.
Hardware
Version
Max Virtual
NICs
4 4
7 10
Type
Max Virtual
Adapters
VMKernel 32
Service
Console
32
Known Issues
VMDirectPathI/O requires GuestOS support. For example, Oplin NIC in
passthrough mode does not perform well with SLES10 in VGT mode.
IPv6 default gateway might not be effective: you might want to use
static routes for the specific destination
Removing IPv4 default gateway might cause IPv6 default gateway to
fail, especially if the gateway does not do IPv6 advertisment.
Configuring a NIC with neither ant static IP (v4 or v6) nor any dynamic
configuration (no DHCP not IPv6 autoconf), after reboot you will have to
remove it and add it again to be able to reconfigure it.

Recovering
Find out the uplink port for the NIC you want to use
esxcfg-vswitch -l
Remove the Uplink from the DVS
esxcfg-vswitch -Q vmnic1 -V 5 dvSwitch
Create a new LifeSaver Standard vSwitch
esxcfg-vswitch a LifeSaver
Give the LifeSaver Standard vSwitch a portgroup and the uplink
esxcfg-vswitch A SOSC LifeSaver
esxcfg-vswitch L vmnic1 LifeSaver
Move the vSwif 0 to the LifeSaver vSwtich
esxcfg-vswif d vswif0
esxcfg-vswif a i DHCP p SOSC vswif0
Use vCenter to fix all via GUI and then cleanup, otherwise:
esxcfg-vswitch -P vmnic1 -V 5 dvSwitch
Questions?

Module 4 - Networking

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Module 4 - Networking

Caricato da

Copyright:

Formati disponibili

vSphere 4.

Potrebbero piacerti anche