UCS Technology Labs VM-FEX on UCS

VM-FEX with DirectPathIO (HighPerformance VM-FEX)

Last updated: April 12, 2013

Task
Create a new high-performance port profile named VM-110-HiPerf in native VLAN 110 that allows
VM-FEX guests to have all networking traffic bypass the ESXi hypervisor.
Make any changes necessary to support VMDirectPathIO in either UCSM or vCenter, ESXi or on
the guest VM itself.
Migrate VM guest machine Win2k8-www-1 to the new VM-FEX DVS and configure for
DirectPathIO High Performance switching.

Configuration

FEEDBACK

Note:
Before we begin, read the following particularly helpful lines from the Cisco VMFEX Using VMware ESX Environment Troubleshooting Guide that go a bit beyond
the scope of what we cover here (because we won't discuss editing conf files
inside the ESXi host) that would be of particular importance if you had a nondefault ESXi installation and customized Tx, Rx, or Completion Queues.
Interrupts

VMDirectPath mode by default Interrupt mode is set to MSI-X which consumes 4 Interrupts
vectors from ESX Host Interrupt pool of 128 maximum limit, if there is a scenario where ESX host
reports Interrupt pool is full and cannot allocate any interrupt vectors to any Virtual Machines' to
change to VMDirectPath mode from emulated mode then will fail and reports in active status in
vCenter on Virtual Machine network properties pane.
Transmit/Receive/Completion Queues

To enable VMDirectPath mode with RSS/multi-queue, you need to ensure that the adapter policy
is configured accordingly. The values can be set in Eth Adapter Policy VMWare PassThru and in
VMXNET3 driver. The values for these queues must meet the following criteria:
The values set for Transmit Queues (TQs) in VMXNET3 should be greater than the
values set for TQs in Eth Adapter Policy VMWare PassThru (largest
VMXNET3.NumTQs on host <= dynamicPolicy.NumTQs).
The values set for Recieve Queues (RQs) in VMXNET3 should be greater than the values
set for RQs in Eth Adapter Policy VMWare PassThru (largest VMXNET3.NumRQs on
host <= dynamicPolicy.NumRQs).
The values set for Transmit Queues (CQs) in VMXNET3 should be greater than the
values set for CQs in Eth Adapter Policy VMWare PassThru (largest
VMXNET3.NumCQs on host <= dynamicPolicy.NumCQs).
The values set for Interrupts (INTRs) in VMXNET3 should be greater than the values set
for INTRs in Eth Adapter Policy VMWare PassThru (largest VMXNET3.NumINTRs on
host <= dynamicPolicy.NumINTRs)
Source: Cisco VM-FEX Using VMware ESX Environment Troubleshooting Guide
(http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns944/basic_troubleshooting_vm_fex.html)

In UCSM, in the left navigation pane, click the Servers tab, navigate to Servers >> Policies >>
Adapter Policies >> Eth Adapter Policy VMWarePassThru. In the right pane, note the Tx,
Rx, and Completion Queues, and modify only if after reading the above note, you know your
ESXi host to be customized (for this lab you may leave them as is). Change the Interrupt Mode
to Msi X and click Save Changes.

Click OK.

Now in the left navigation pane, click the Servers tab and navigate to Servers >> Service
Profiles >> root org >> Sub-Organizations >> CUST2 >> ESXi-VM-FEX-iSCSI-1. In the right
pane, click the Policies tab and expand BIOS Policy. Note the BIOS policy in use. We may
not need to modify that policy, but we do need to check it to ensure proper configuration for
usage with VM-FEX and DirectPathIO.

Navigate to Servers >> Policies >> BIOS Policies >> Loud-w-VT. In the right pane, click the
Advanced tab, and then click the Processor tab. Notice that Virtualization Technology is
enabled. This is actually the default for a B22-M3 blade, but we had to be absolutely certain that
it was enabled to support DirectPathIO in VM-FEX.

In the left navigation pane, click the VM tab, right-click Port Profiles, and click Create Port
Profile.

Enter the name as shown, choose the native VLAN 110, and select the Host Network IO
Performance mode of High Performance. Click OK.

Click OK.

Click the new port profile in the left navigation pane, and in the right pane click Profile Clients.

Create a new profile client with relatively the same name that pushes the port profile to the VMFEX DVS.

Click OK.

Back in vCenter, right-click the Win2k8-www-1 guest and click Edit Settings.

On the Hardware tab, ensure that the Adapter Type is VMXNET 3. Change the port group to
our new VM-110-HiPerf (VM-FEX-DVS1). Note that in the DirectPath I/O section it tells us that
it is Inactive because (assuming we have the proper adapter type) all memory must be reserved
(it cannot be shared with any other guests).

Note:
If the current network adapter is not the type of VMXNET 3, take time now to
delete the current adapter, add a new network adapter with this specific adapter
type, console into the guest, install VM Tools (which contain the driver), and
reboot the guest VM. Failure to use the proper network driver type will cause an
inability utilize VMDirectPathIO.

Click the Resources tab. Click Memory. Select the Reserve all guest memory (All locked)
check box. Click OK.

Right-click the Win2k8-www-1 guest again and click Edit Settings.

Notice now that if you followed all steps carefully, VM DirectPath I/O is now Active. You have
successfully bypassed the hypervisor for all network switching. vMotion is even supported with
this mode enabled, but only with Cisco UCS.

Back in UCSM, on the VM tab, notice that the new Virtual Machine appears in the DVS, and
that it has two vNICs. This is expected if you had migrated this guest to the VM-FEX DVS
PassThru Switching mode prior to migrating it to DirectPathIO mode. Only the latter vNIC is
actually active now, and the other is a mere remnant.

From Server1 connected to N5K1, try to ping the Win2k8-www-1 guest at the IP of 10.0.110.211.
We should see that it is successful.

In NX-OS on UCS-FI-A, notice that we have two vNICs for this guest, but that the former (PTS
vNIC) is nonParticipating.

INE-UCS-01-A(nxos)# sh int br | in Veth

Vethernet

VLAN

Type Mode

Status Reason

Speed

Veth709

eth trunk up

none

auto

Veth711

eth trunk up

none

auto

Veth713

eth trunk up

none

auto

Veth716

eth trunk up

none

auto

Veth717

117

eth trunk up

none

auto

Veth32769

110

eth trunk up

none

auto

Veth32770

110

eth trunk down

nonPartcipating

auto

Veth32771

110

eth trunk up

none

auto

INE-UCS-01-A(nxos)#

Note the same thing in NX-OS on UCS-FI-B.

INE-UCS-01-B(nxos)# sh int br | in Veth

Vethernet

VLAN

Type Mode

Status Reason

Speed

Veth710

eth trunk up

none

auto

Veth712

eth trunk up

none

auto

Veth714

eth trunk up

none

auto

Veth715

eth trunk up

none

auto

Veth718

117

eth trunk up

none

auto

Veth32769

110

eth trunk up

none

auto

Veth32770

110

eth trunk down

nonPartcipating

auto

Veth32771

110

eth trunk up

none

auto

INE-UCS-01-B(nxos)#

^ back to top

Disclaimer (http://www.ine.com/feedback.htm) | Privacy Policy (http://www.ine.com/resources/)

Inc., All Rights Reserved (http://www.ine.com/about-us.htm)

2013 INE