Hyper-V VMs losing network

connectivity? Beware VMQ…

http://blog.insynctechnology.com.au/hyper-v-vms-losing-network-connectivity-beware-vmq

Consider the following scenario:

You’ve deployed your Virtual Machines on Hyper-V hosts that are running Windows Server
2012 or 2012 R2. Everything appears to be running swimmingly. However, you soon start
to experience the following symptoms:

 Virtual machines randomly lose network connectivity. The network adapter appears to be
working in the virtual machine. However, you cannot ping or access network resources
from the virtual machine. Restarting the virtual machine does not resolve the issue
 You cannot ping or connect to a virtual machine from a remote computer, you are only
able to connect via the Hyper-V console

When this occurs, the only fix is to restart the Hyper-V host. Restarting the VM does not
resolve the issue. This also does not address the underlying issue as to why this is occurring
in the first place.

VMQ

This issue has been around for sometime, with Microsoft acknowledging and discussing
ithere. However, this only refers to the “NetXtreme 1-gigabit network adapters”, which in
my case were not being used. Looking a little deeper, the issue affected a broader number
of network adapters, all of which used common components manufactured by a company
called Emulex. If you were using an affected adapter on a Windows Server 2012 or 2012
R2 Hyper-V host, and you had Virtual Machine Queue (VMQ) enabled, you could
potentially see this issue.

VMQ is only relevant if the network adapters used on your Hyper-V hosts are 10-gigabit or
faster. In my case, all adapters were 1-gigabit, meaning I could simply disable VMQ and
see if that resolved my problem. To check whether or not VMQ is enabled, run the
following:
Get-NetAdapterVmq

To disable:

Set-NetAdaptervmq -Name “Embedded LOM 1 Port 2” -Enabled $false

Alternatively, you can also disable from Device Manager:

In Spanish:

Once you have disabled VMQ on the adapter, make sure you also disable it from the
Hardware Acceleration menu from Setting on all Virtual Machines
This resolved the issue for me.
How to Disable VMQ with PowerShell
https://community.spiceworks.com/how_to/124202-how-to-disable-vmq-with-powershell

VMQ can cause erratic virtual machine network performance when enabled, such as
latency and dropped network connections. This is especially common with Broadcom
Gigabit network adapters.

This how-to will show you how to disable VMQ on your Windows host server using
PowerShell.

2 Steps total
Step 1: Find the current VMQ status on your NIC's.

On your host, open up a PowerShell prompt as an Administrator and enter the following
command to return all of your installed NIC's and their VMQ status:

Get-NetAdapterVmq

Step 2: Set VMQ to Disabled

Next, for any NIC listed in step 1 above with VMQ set to $true, run the below command to
disable VMQ. The NIC will become disconnected for approximately one second after you
run this command:

Set-NetAdapterVmq -Name "NIC Name" -Enabled $False

Poor network performance on virtual
machines on a Windows Server 2012
Hyper-V host if VMQ is enabled
https://support.microsoft.com/en-ca/help/2902166/poor-network-performance-on-virtual-
machines-on-a-windows-server-2012

Symptoms

Consider the following scenario:

 You have a Windows Server 2012 Hyper-V host that uses a Broadcom network adapter.
 You create a virtual switch that binds to the network adapter.
 You enable virtual machine queue (VMQ) on the Hyper-V host.

In this scenario, you experience poor network performance on the virtual machines that are
hosted on the Hyper-V host.
Cause

This is a known issue with Broadcom network adapter drivers when VMQ is enabled. The following
Broadcom network adapters are affected:

 57712
 57800
 57810
 57840

Resolution

Broadcom is aware of this issue and will release a driver update to resolve the issue. Until the
driver update is available, you can work around the issue by taking one of the following actions:

 Disable VMQ on the Virtual Network Adapter by using the following Windows PowerShell
cmdlet:

Set-VMNetworkAdapter –ManagementOS -Name

<VirtualNetworkAdapterName> -VmqWeight 0
 Note If the problem is not solved, disable VMQ on the Broadcom network adapter.

 Change the Media Access Control (MAC) address of the virtual switch.

Note IP address of the Hyper-V host may change if DHCP is used.

If you use System Center Virtual machine Manager (SCVMM), SCVMM can automatically
assign a new MAC address.

To manually assign a MAC address, follow these steps:

1. Assign a MAC address by using the following Windows PowerShell cmdlet:

Set-VMNetworkAdapter –ManagementOS -Name

<VirtualNetworkAdapterName> -StaticMacAddress <MacAddress>
2. Assign a dynamically generated MAC address by using the following Windows
PowerShell cmdlet:

Set-VMNetworkAdapter –ManagementOS -Name

<VirtualNetworkAdapterName> -DynamicMacAddress
 If there is another unaffected network adapter on the Hyper-V host, bind the virtual switch
to that network adapter.
Hyper-V: Virtual Machine Queues (VMQ)
FAQs

https://portal.nutanix.com/#page/kbs/details?targetId=kA0600000008eNTCAY

Versions affected
AOS 4.5 AOS 3.5.x

Description
What is VMQ?

Virtual Machine Queue (VMQ) is a type of hardware packet filtering that delivers packet data from an outside
virtual machine network directly to a virtualized host operating system. VMQ works by allowing a Network
Interface Controller (NIC) to transfer incoming frames directly to the receive buffer of the NIC by using Direct
Memory Access (DMA). This simplifies the overhead involved in routing packets and reduces the need for
copying them from the management operating system (OS) to the virtual machine (VM). Instead of one
processor handling all of the network data exchanges, different processors can process packets for different
virtual machines with VMQ. This eliminates the dependence on driver-based traffic exchanges and improves
the transfer of network traffic.

Note: VMQ support was introduced in AOS version 4.5, and must only be enabled if the Intel driver is running
the version described in the Solution section of the following article.

How to enable or disable VMQ?

Enable or disable VMQ by using the System Center Virtual Machine Management (SCVMM) utility or
PowerShell commands.

Note: Enabling VMQ might cause a short network disruption and you must enable VMQ during a maintenance
window.

Using SCVMM

Following is the procedure to enable VMQ using SCVMM.

1. Select the VM.

2. Go to Settings > Hardware > Network Adapter > Hardware Acceleration.
3. Select Enable virtual machine queue in the Hardware Acceleration settings dialog box.
Using PowerShell
Following is the PowerShell command to enable VMQ.

Enable-NetAdapterVmq

Solution
Updating the NX Series Intel 10 GB Driver to Enable VMQ

Download the current driver version (the network driver that is qualified to have VMQ enabled has a release
date of July 7, 2014, or later).

1. Log on to the host by using Remote Desktop or Remote PowerShell.

2. Run the following PowerShell command.

Get-NetAdapter | where InterfaceDescription -like "*10 Gigabit*" | fl Name,

InterfaceDescription, DriverInformation

Downloading the Intel driver to a share or directory local to each node

1. Download the Intel NIC driver from the following location.

https://downloadcenter.intel.com/download/23073/Network-Adapter-Driver-for-Windows-Server-2012-R2-
2. Extract the contents of the executable that you downloaded.
 3. Copy the following relative directory to a shared location addressable by each node, or locally to each node
you want to upgrade.

Extract_Directory\PROWinx64\PROXGB\Winx64\NDIS64

Installing the driver

Use the PnPUtil command to add (-a) and install (-i) the driver.

Pnputil –i –a \\hvtest-smb\hvtest-library\NDIS64\*.inf

Validating driver version after the installation

Run the following command after the driver installation is complete.

Get-NetAdapter | where InterfaceDescription -like "*10 Gigabit*" | fl Name,

InterfaceDescription, DriverInformation

Checking VMQ Setting

Verify that the driver installation enables VMQ by default.

Get-NetAdapterVMQ | where InterfaceDescription -like "*10 Gigabit*”

To verify, VMQ is being used by the VMs.

allssh "winsh 'get-vm | Get-VMNetworkAdapter | select VMName,VMQUsage'"

Example Output
Where the host is node4, CVM-D, and a VM Xen, notice the queue ID and processor core assignment.

================== x.x.x.x =================

Name QueueID MacAddress VlanID Processor

VmFriendlyName
---- ------- ---------- ------ ---------
--------------

Ethernet 3 0 0:0 NODE4

Ethernet 3 1 00-15-5D-7E-78-0A 0:18 NTNX-
xxxx-D-CVM
Ethernet 3 2 00-15-5D-7E-78-18 0:18 Xen
Ethernet 0 0:0 NODE4
Ethernet 1 0C-C4-7A-4C-6B-5F 0:2 NODE4
NetAdapterTeam 0 0:0 NODE4
NetAdapterTeam 1 0C-C4-7A-4C-6B-5F 0:2 NODE4
NetAdapterTeam 2 00-15-5D-7E-78-0A 0:18 NTNX-
xxxx-D-CVM
NetAdapterTeam 4 00-15-5D-7E-78-18 0:18 Xen

Tags

Networking;Troubleshooting;Performance;Hyper V;Hardware;Resiliency