Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Paul Meehan
http://paulpmeehan.com
Introduction ....................................................................................................................... 6
1.1
1.2
2.3
2.4
Hardware Requirements for vCenter Server, vCenter Single Sign On, vSphere
Client, and vSphere Web Client ............................................................................................ 11
2.5
vCenter Server and vSphere Client System Recommendations for Performance
Based on Deployment Size ................................................................................................... 14
2.6
2.7
2.8
Prepare Your System and Install the Auto Deploy Server ........................................ 19
2.9
2.10
2.11
2.12
HA Clusters.......................................................................................................................30
3.1
3.2
3.3
3.4
3.5
Networking ....................................................................................................................... 49
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
Storage .............................................................................................................................. 61
5.1
5.2
5.3
5.4
5.5
5.6
iSCSI .......................................................................................................................... 67
5.7
5.8
5.9
5.10
5.11
5.12
5.13
5.14
5.15
5.16
5.17
5.18
5.19
5.20
5.21
5.22
5.23
5.24
5.25
5.26
5.27
5.28
5.29
5.30
5.31
5.32
5.33
5.34
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
6.10
6.11
6.12
6.13
6.14
6.15
6.16
6.17
6.18
6.19
6.20
7.3
7.4
7.5
7.6
7.7
7.8
7.9
7.10
7.11
7.12
7.13
7.14
7.15
7.16
Best Practices for Virtual Machine and Host Security ............................................ 190
7.17
7.18
7.19
7.20
7.21
7.22
7.23
7.24
8.3
8.4
9.2
9.3
9.4
Change CPU Hot Plug Settings in the vSphere Web Client ................................... 209
9.5
9.6
9.7
Configure Fibre Channel NPIV Settings in the vSphere Web Client ...................... 219
9.8
Managing Multi-Tiered Applications with vSphere vApp in the vSphere Web Client
221
9.9
9.10
9.11
9.12 Change Disk Mode to Exclude Virtual Disks from Snapshots in the vSphere Web
Client 229
1 Introduction
Hello and Welcome. Its 02/01/2014.
While studying for VCAP-DCD and preparing a submission for VCDX, Ive been trying to
find a really good source for all pre-requisites, best practices, requirements and other
important information used by vSphere Architects, admins and end users. As a designer,
understanding the impact of your decisions against these items is key.
I like to have things in one place if possible. This document has been created from the
existing vSphere official documentation set (pubs.vmware.com), using copy and paste, to
capture the requirements above into a single document that can be used as a reference.
Note:
This is the result of my personal editing of what I believe are the useful
nuggets we all need to know. However, once I started I realised it could
never be a 3-4 page document which was my original intention, so better
to make it complete and use an index to allow people to search around.
The Intellectual Property for this material is not mine. It has not been
added to in ANY way by me. This material is VMware Copyrighted
material that is freely available on pubs.vmware.com.
Once I started doing this I noticed this link where this is a massive
amount of documentation, best practices etc. http://bit.ly/1caCcQs.
o That should be Bookmark #1 for any vSphere/vCloud Architect.
I have copied what I believe are other nuggets of info that will help you
implementing the best possible vSphere design.
The current version (1.0) relates to v5.1 which is the subject of my studies.
As with all things in the Vmware community its important to pass this around so folks
following the same certification route might find it useful, or just for general reference. So
please share and I truly hope you find it useful.
This document assumes a medium level of understanding and is not a build guide.
Please email me on info@tieroneconsulting.ie with any feedback or other ideas.
Date
02/01/2014
Author
Paul P Meehan
Description
Issued version for vSphere 5.1.
Still requires some additional input from other
vSphere official documentation.
Availability
Networking
Storage
Security
Resource Management
Installation and Setup
Host Profiles
Virtual Machine Administration
To install and use ESXi 5.1, your hardware and system resources must meet the following
requirements:
Supported server platform. For a list of supported platforms, see the VMware
Compatibility Guide at http://www.vmware.com/resources/compatibility.
ESXi 5.1 will install and run only on servers with 64-bit x86 CPUs.
ESXi 5.1 requires a host machine with at least two cores.
ESXi 5.1 supports only LAHF and SAHF CPU instructions.
ESXi 5.1 requires the NX/XD bit to be enabled for the CPU in the BIOS.
ESXi supports a broad range of x64 multicore processors. For a complete list of
supported processors, see the VMware compatibility guide at
http://www.vmware.com/resources/compatibility.
ESXi requires a minimum of 2GB of physical RAM. Provide at least 8GB of RAM to
take full advantage of ESXi features and run virtual machines in typical production
environments.
To support 64-bit virtual machines, support for hardware virtualization (Intel VT-x
or AMD RVI) must be enabled on x64 CPUs.
One or more Gigabit or 10Gb Ethernet controllers. For a list of supported network
adapter models, see the VMware Compatibility Guide at
http://www.vmware.com/resources/compatibility.
Any combination of one or more of the following controllers:
o Basic SCSI controllers. Adaptec Ultra-160 or Ultra-320, LSI Logic FusionMPT, or most NCR/Symbios SCSI.
o RAID controllers. Dell PERC (Adaptec RAID or LSI MegaRAID), HP Smart
Array RAID, or IBM (Adaptec) ServeRAID controllers.
SCSI disk or a local, non-network, RAID LUN with unpartitioned space for the virtual
machines.
For Serial ATA (SATA), a disk connected through supported SAS controllers or
supported on-board SATA controllers. SATA disks will be considered remote, not
local. These disks will not be used as a scratch partition by default because they are
seen as remote.
Note
You cannot connect a SATA CD-ROM device to a virtual machine on an ESXi 5.1 host. To use
the SATA CD-ROM device, you must use IDE emulation mode.
Network booting or provisioning with VMware Auto Deploy requires the legacy BIOS
firmware and is not available with UEFI.
ESXi can boot from a disk larger than 2TB provided that the system firmware and the
firmware on any add-in card that you are using support it. See the vendor documentation.
Note
Changing the boot type from legacy BIOS to UEFI after you install ESXi 5.1 might cause the
host to fail to boot. In this case, the host displays an error message similar to: Not a
VMware boot bank. Changing the host boot type between legacy BIOS and UEFI is not
supported after you install ESXi 5.1.
System ElementRecommendation
ESXi hosts require more RAM than typical servers. Provide at least 8GB
of RAM to take full advantage of ESXi features and run virtual machines
in typical production environments. An ESXi host must have sufficient
RAM to run concurrent virtual machines. The following examples are
provided to help you calculate the RAM required by the virtual machines
running on the ESXi host.
Operating four virtual machines with Red Hat Enterprise Linux or
Windows XP requires at least 3GB of RAM for baseline performance. This
figure includes approximately 1024MB for the virtual machines, 256MB
minimum for each operating system as recommended by vendors.
RAM
Running these four virtual machines with 512MB RAM requires that the
ESXi host have approximately 4GB RAM, which includes 2048MB for the
virtual machines.
These calculations do not take into account possible memory savings from
using variable overhead memory for each virtual machine. See vSphere
Resource Management .
Dedicated Fast
Ethernet adapters
for virtual
machines
Disk location
Place all data that your virtual machines use on physical disks allocated
specifically to virtual machines. Performance is better when you do not
place your virtual machines on the disk containing the ESXi boot image.
Use physical disks that are large enough to hold disk images that all the
virtual machines use.
VMFS5
partitioning
The ESXi installer creates the initial VMFS volumes on the first blank
local disk found. To add disks or modify the original configuration, use
the vSphere Client. This practice ensures that the starting sectors of
partitions are 64K-aligned, which improves storage performance.
For AMD Opteron-based systems, the processors must be Opteron Rev E or later.
For Intel Xeon-based systems, the processors must include support for Intel
Virtualization Technology (VT). Many servers that include CPUs with VT support
might have VT disabled by default, so you must enable VT manually. If your CPUs
support VT ,but you do not see this option in the BIOS, contact your vendor to request
a BIOS version that lets you enable VT support.
To determine whether your server has 64-bit VMware support, you can download the CPU
Identification Utility from the VMware Web site.
Intel or AMD x64 processor with two or more logical cores, each with a
speed of 2GHz.
Memory
Disk storage
Network speed
1Gbps
Requirement
Processor
Intel or AMD x64 processor with two or more logical cores, each with a
speed of 2GHz.
Memory
Disk storage
Network speed
At least 60GB for medium- to large-sized inventories (more than 100 hosts
or 1000 virtual machines).
If vCenter Inventory Service runs on the same host machine as vCenter
Server, see Minimum Hardware Requirements for vCenter Server.
1Gbps
Processor
Memory
Requirement
Driven Storage Service. When you install vCenter Server, you select the size
of your vCenter Server inventory to allocate memory for these services. The
inventory size determines the maximum JVM heap settings for the
services. You can adjust this setting after installation if the number of hosts
in your environment changes. See the recommendations in JVM Heap
Settings for vCenter Server.
The amount of disk storage needed for the vCenter Server installation
depends on your vCenter Server configuration.
Disk storage
The JVM heap settings for vCenter Server depend on your inventory size. See Configuring
VMware Tomcat Server Settings in vCenter Server 5.1.
Profile-Driven
Storage
Service
3GB
512MB
6GB
1GB
12GB
2GB
vCenter Server
Inventory
VMware VirtualCenter
Management Webservices
(tc Server)
2GB
Note
Installing vCenter Server on a network drive or USB flash drive is not supported.
For the hardware requirements of your database, see your database documentation. The
database requirements are in addition to the vCenter Server requirements if the database
and vCenter Server run on the same machine.
2.4.5 VMware vCenter Server Appliance Hardware Requirements and
Recommendations
Important
The embedded database is not configured to manage an inventory that contains
more than 5 hosts and 50 virtual machines. If you use the embedded database
with the vCenter Server Appliance, exceeding these limits can cause numerous
problems, including causing vCenter Server to stop responding.
Log
Management Agent (hostd)
VirtualCenter Agent (vpxa)
vSphere HA agent (Fault
Domain Manager, fdm)
10
50MB
Important
The recommended disk sizes assume default log levels. If you configure more detailed log
levels, more disk space is required.
Cores
Memory
Disk
vCenter Server
4GB
5GB
vSphere Client
1GB
1.5GB
Cores
Memory
Disk
vCenter Server
8GB
10GB
vSphere Client
1GB
1.5GB
Cores
Memory
Disk
vCenter Server
16GB
10GB
vSphere Client
1GB
1.5GB
At least 2GB: 1GB for the Java heap, and 1GB for
The resident code
Disk Storage
Networking
Log
10240KB
10
100MB
5120KB
10
50MB
5120KB
10
50MB
Description
vCenter Server requires port 80 for direct HTTP connections. Port 80
redirects requests to HTTPS port 443. This redirection is useful if you
accidentally use http://server instead of https://server.
80
If you use a custom Microsoft SQL database (not the bundled SQL Server
2008 database) that is stored on the same host machine as the vCenter
Server, port 80 is used by the SQL Reporting Service. When you install
vCenter Server, the installer will prompt you to change the HTTP port for
vCenter Server. Change the vCenter Server HTTP port to a custom value to
ensure a successful installation.
Microsoft Internet Information Services (IIS) also use port 80. See Conflict
Between vCenter Server and IIS for Port 80.
389
This port must be open on the local and all remote instances of vCenter
Server. This is the LDAP port number for the Directory Services for the
Port
Description
vCenter Server group. The vCenter Server system needs to bind to port
389, even if you are not joining this vCenter Server instance to a Linked
Mode group. If another service is running on this port, it might be
preferable to remove it or change its port to a different port. You can run
the LDAP service on any port from 1025 through 65535.
If this instance is serving as the Microsoft Windows Active Directory,
change the port number from 389 to an available port from 1025 through
65535.
The default port that the vCenter Server system uses to listen for
connections from the vSphere Client. To enable the vCenter Server system
to receive data from the vSphere Client, open port 443 in the firewall.
443
The vCenter Server system also uses port 443 to monitor data transfer from
SDK clients.
If you use another port number for HTTPS, you must use ip-address:port
when you log in to the vCenter Server system.
636
For vCenter Server Linked Mode, this is the SSL port of the local instance.
If another service is running on this port, it might be preferable to remove
it or change its port to a different port. You can run the SSL service on any
port from 1025 through 65535.
Description
80
443
The vCenter Server system also uses port 443 to monitor data transfer
from SDK clients.
If you use another port number for HTTPS, you must use ipaddress:port when you log in to the vCenter Server system.
902
The default port that the vCenter Server system uses to send data to
managed hosts. Managed hosts also send a regular heartbeat over UDP
port 902 to the vCenter Server system. This port must not be blocked
Port
Description
by firewalls between the server and the hosts or between hosts.
Port 902 must not be blocked between the vSphere Client and the
hosts. The vSphere Client uses this port to display virtual machine
consoles.
8080
8443
10080
10443
10109
514
2.8 Prepare Your System and Install the Auto Deploy Server
Before you turn on a host for PXE boot with vSphere Auto Deploy, you must install
prerequisite software and set up the DHCP and TFTP servers that Auto Deploy interacts
with.
Ensure that the hosts that you will provision with Auto Deploy meet the hardware
requirements for ESXi 5.1.
Note
You cannot provision EFI hosts with Auto Deploy unless you switch the EFI system to BIOS
compatibility mode.
Ensure that the ESXi hosts have network connectivity to vCenter Server and that all
port requirements are met.
If you want to use VLANs in your Auto Deploy environment, you must set up the end
to end networking properly. When the host is PXE booting, the UNDI driver must be
set up to tag the frames with proper VLAN IDs. You must do this set up manually by
making the correct changes in the BIOS. You must also correctly configure the ESXi
port groups with the correct VLAN IDs. Ask your network administrator how VLAN
IDs are used in your environment.
Ensure that you have enough storage for the Auto Deploy repository. The Auto
Deploy server uses the repository to store data it needs, including the rules and rule
sets you create and the VIBs and image profiles that you specify in your rules.
Best practice is to allocate 2GB to have enough room for four image profiles and some
extra space. Each image profile requires approximately 350MB. Determine how
much space to reserve for the Auto Deploy repository by considering how many
image profiles you expect to use.
Obtain the vCenter Server installation media, which include the Auto Deploy
installer, or deploy the vCenter Server Appliance.
Secure your network as you would for any other PXE-based deployment method.
Auto Deploy transfers data over SSL to prevent casual interference and snooping.
However, the authenticity of the client or the Auto Deploy server is not checked
during a PXE boot. .
Note
Auto Deploy is not supported with NPIV (N_Port ID Virtualization).
Set up a remote Syslog server. See the vCenter Server and Host Management
documentation for Syslog server configuration information. Configure the first host
you boot to use the remote syslog server and apply that host's host profile to all other
target hosts. Optionally, install and use the vSphere Syslog Collector, a vCenter
Server support tool that provides a unified architecture for system logging and
enables network logging and combining of logs from multiple hosts.
Install ESXi Dump Collector and set up your first host so all core dumps are directed
to ESXi Dump Collector and apply the host profile from that host to all other hosts.
See Configure ESXi Dump Collector with ESXCLI and Set Up ESXi Dump Collector
from the Host Profiles Interface in the vSphere Client.
Auto Deploy does not support a pure IPv6 environment because the PXE boot
specifications do not support IPv6. However, after the initial PXE boot state, the rest
of the communication can happen over IPv6. You can register Auto Deploy to the
vCenter Server system with IPv6, and you can set up the host profiles to bring up
hosts with IPv6 addresses. Only the initial boot process requires an IPv4 address.
Follow best practices when installing vSphere Auto Deploy and when using Auto Deploy
with other vSphere components. Set up a highly available Auto Deploy infrastructure in large
production environments or when using stateless caching. Follow all security guidelines that
you would follow in a PXE boot environment, and consider the recommendations in this
chapter.
2.9.1 Auto Deploy and vSphere HA Best Practices
You can improve the availability of the virtual machines running on hosts provisioned with
Auto Deploy by following best practices.
Some environments configure the hosts provisioned with Auto Deploy with a
distributed switch or configure virtual machines running on the hosts with Auto Start
Manager. In those environments, deploy the vCenter Server system so that its
availability matches the availability of the Auto Deploy server. Several approaches are
possible.
o
o
In a proof of concept environment, deploy the vCenter Server system and the
Auto Deploy server on the same system. In all other situations, install the two
servers on separate systems.
Deploy vCenter Server Heartbeat.
VMware vCenter Server Heartbeat delivers high availability for VMware
vCenter Server, protecting the virtual and cloud infrastructure from
application, configuration, operating system, or hardware related outages.
Deploy the vCenter Server system in a virtual machine. Run the vCenter Server virtual
machine in a vSphere HA enabled cluster and configure the virtual machine with a vSphere
HA restart priority of high. Include two or more hosts in the cluster that are not managed by
Auto Deploy and pin the vCenter Server virtual machine to these hosts by using a rule
(vSphere HA DRS required VM to host rule). You can set up the rule and then disable DRS if
you do not wish to use DRS in the cluster. The greater the number of hosts that are not
managed by Auto Deploy the greater your resilience to host failures.
Note
This approach is not suitable if you use Auto Start Manager because Auto Start Manager is
not supported in a cluster enabled for vSphere HA.
VLAN
Using Auto Deploy in environments that do not use VLANs is highly
Considerations recommended.
If you intend to use Auto Deploy in an environment that uses VLANs, you
must make sure that the hosts you want to provision can reach the DHCP
server. How hosts are assigned to a VLAN depends on the setup at your site.
The VLAN ID might be assigned by the switch or by the router, or you
might be able to set the VLAN ID in the host's BIOS or through the host
profile. Contact your network administrator to determine the steps for
allowing hosts to reach the DHCP server.
2.9.2.1 Auto Deploy and VMware Tools Best Practices
See the VMware Knowledge Base article 2004018 for Auto Deploy and VMware Tools best
practices.
vCenter Server system. This IP address must have a valid (internal) domain name
system (DNS) registration. Ensure that the ESXi host management interface has a
valid DNS resolution from the vCenter Server and all vSphere Clients. Ensure that the
vCenter Server has a valid DNS resolution from all ESXi hosts and all vSphere
Clients. If you use DHCP instead of a static IP address for vCenter Server, make sure
that the vCenter Server computer name is updated in the domain name service
(DNS). Ping the computer name to test this connection. For example, if the computer
name is host-1.company.com, run the following command in the Windows command
prompt:
ping host-1.company.com
If you can ping the computer name, the name is updated in DNS.
For the vCenter Single Sign-On installer to automatically discover Active Directory identity
sources, verify that the following conditions are met.
The Active Directory identity source must be able to authenticate the user who is
logged in to perform the Single Sign-On installation.
The DNS of the Single Sign-On Server host machine must contain both lookup and
reverse lookup entries for the domain controller of the Active Directory. For example,
pinging mycompany.com should return the domain controller IP address for
mycompany. Similarly, the ping -a command for that IP address should return the
domain controller hostname. Avoid trying to correct name resolution issues by
editing the hosts file. Instead, make sure that the DNS server is correctly set up.
The system clock of the Single Sign-On Server host machine must be synchronized
with the clock of the domain controller.
Verify that your vCenter Server database meets the database requirements. See
vCenter Server Database Configuration Notes and Preparing vCenter Server
Databases.
Create a vCenter Server database, unless you plan to install the bundled database.
Create a vCenter Single Sign-On database, unless you plan to install the bundled
database.
If you are using an existing database for Single Sign On, you must create a database
user (RSA_USER) and database administrator (RSA_DBA) to use for the Single Sign
On database installation and setup. To create these users, run the script
rsaIMSLiteDBNameSetupUsers.sql. The script is included in the vCenter Server
installer download package, at vCenter Server Installation directory\SSOServer.
If you are using an existing database with your vCenter Single Sign-On installation or
upgrade, make sure that the table spaces are named RSA_DATA and RSA_INDEX.
Any other table space names will cause the vCenter Single Sign-On Installation to fail.
If you are using an existing database for Single Sign-On, to ensure that table space is
created for the database, run the script rsaIMSLite<DBName>SetupTablespaces.sql.
The script is included in the vCenter Server installer download package, at vCenter
Server Installation directory\Single Sign
On\DBScripts\SSOServer\Schema\your_existing_database. You can run this script
prior to the installation, or during the installation, when you are prompted by the
installer. You can leave the installer to run the script, and resume the installer after
you run the script.
2.11.1 vCenter Single Sign On Components
vCenter Single Sign On includes these components: STS (Security Token Service), an
administration server, vCenter Lookup Service, and the RSA SSPI service.
When you install vCenter Single Sign-On, the following components are deployed.
STS (Security
Token Service)
Administration
server
vCenter Lookup
Service
RSA SSPI service The Security Support Provider Interface is a Microsoft Windows-based
API used to perform authentication against Security Support Providers
such as NTLM and Kerberos.
2.11.2 vCenter Lookup Service
vCenter Lookup Service is a component of vCenter Single Sign On. Lookup Service registers
the location of vSphere components so they can securely find and communicate with each
other.
The vCenter Single Sign-On installer also deploys the VMware Lookup Service on the same
address and port. The Lookup Service enables different components of vSphere to find one
another in a secure way. When you install vCenter Server components after vCenter Single
Sign-On, you must provide the Lookup Service URL. The Inventory Service and the vCenter
Server installers ask for the Lookup Service URL and then contact the Lookup Service to find
vCenter Single Sign-On. After installation, the Inventory Service and vCenter Server are
registered in Lookup Service so other vSphere components, like the vSphere Web Client, can
find them.
2.11.3 Setting the vCenter Server Administrator User
In vCenter Server 5.1 with vCenter Single Sign On, the way you set the vCenter Server
administrator user depends on your vCenter Single Sign On deployment.
In vSphere versions before vSphere 5.1, vCenter Server administrators are the users that
belong to the local operating system administrators group.
In vSphere 5.1, when you install vCenter Server, you must provide the default (initial)
vCenter Server administrator user or group. For small deployments where vCenter Server
and vCenter Single Sign-On are deployed on the same host machine, you can designate the
local operating system group Administrators as vCenter Server administrative users. This
option is the default. This behavior is unchanged from vCenter Server 5.0.
For larger installations, where vCenter Single Sign-On and vCenter Server are deployed on
different hosts, you cannot preserve the same behavior as in vCenter Server 5.0. Instead,
assign the vCenter Server administrator role to a user or group from an identity source that
is registered in the vCenter Single Sign-On server: Active Directory, OpenLDAP, or the
system identity source.
2.11.4 Authenticating to the vCenter Server 5.1 Environment
In vCenter Server 5.1, users authenticate through vCenter Single Sign On.
In vCenter Server versions earlier than vCenter Server 5.1, when a user connects to vCenter
Server, vCenter Server authenticates the user by validating the user against an Active
Directory domain or the list of local operating system users.
Because vCenter Server now has its own vCenter Single Sign-On server, you must create
Single Sign-On users to manage the Single Sign-On server. These users might be different
from the users that administer vCenter Server.
The default vCenter Single Sign-On administrator user ID is admin@System-Domain. You
can create Single Sign-On administrator users with the Single Sign-On administration tool in
the vSphere Web Client. You can associate the following permissions with these users: Basic,
Regular, and Administrator.
Users can log in to vCenter Server with the vSphere Client or the vSphere Web Client.
Using the vSphere Client, the user logs in to each vCenter Server separately. All
linked vCenter Server instances are visible on the left pane of the vSphere Client. The
vSphere Client does not show vCenter Server systems that are not linked to the
vCenter Server that the user logged in to unless the user connects to those vCenter
Server systems explicitly. This behavior is unchanged from vCenter Server versions
earlier than version 5.1.
Using the vSphere Web Client, users authenticate to vCenter Single Sign-On, and are
connected to the vSphere Web Client. Users can view all the vCenter Server instances
that the user has permissions on. After users connect to vCenter Server, no further
authentication is required. The actions users can perform on objects depend on the
user's vCenter Server permissions on those objects.
For vCenter Server versions earlier than vCenter Server 5.1, you must explicitly register each
vCenter Server system with the vSphere Web Client, using the vSphere Web Client
Administration Application.
For more information about vCenter Single Sign On, see vSphere Security.
2.11.5 How vCenter Single Sign-On Deployment Scenarios Affect Log In
Behavior
The way that you deploy vCenter Single Sign-On and the type of user who installs vCenter
Single Sign-On affects which administrator user accounts have privileges on the Single SignOn server and on vCenter Server.
During the vCenter Server installation process, certain users are granted privileges to log in
to vCenter Server and certain users are granted privileges to manage vCenter Single Sign-On.
The vCenter Server administrator might not be the same user as the vCenter Single Sign-On
administrator. This means that when you log in to the vSphere Web Client as the default
Single Sign-On administrator (admin@System-Domain), you might not see any vCenter
Server systems in the inventory. The inventory appears to be empty because you see only the
systems upon which you have privileges in the vSphere Web Client.
This also means that when you log in to the vSphere Web Client as the default vCenter Server
administrator, you might not see the vCenter Single Sign-On configuration tool. The
configuration tool is not present because only the default vCenter Single Sign-On
Administrator (admin@System-Domain) is allowed to view and manage vCenter Single SignOn after installation. The Single Sign-On administrator can create additional administrator
users if necessary.
2.11.6 Login Behavior When You Use vCenter Simple Install
The vCenter Simple Install process installs vCenter Single Sign-On, the Inventory Service,
and vCenter Server on one system. The account you use when you run the Simple Install
process affects which users have privileges on which components.
When you log in as a domain account user or local account user to install vCenter Server
using vCenter Simple Install, the following behavior occurs upon installation.
By default, users in the local operating system Administrators group can log in to the
vSphere Web Client and vCenter Server. These users cannot configure Single Sign-On
or view the Single Sign-On management interface in the vSphere Web Client.
By default, the vCenter Single Sign-On administrator user is admin@SystemDomain. This user can log in to the vSphere Web Client to configure Single Sign-On
and add accounts to manage Single Sign-On if necessary. This user cannot view or
configure vCenter Server.
If you are logged in as a domain account user, the default Active Directory identity
sources are discovered automatically during vCenter Single Sign On installation. If
you are logged in as a local account user, Active Directory identity sources are not
discovered automatically during vCenter Single Sign On installation.
The local operating system (localos or hostname) users are added as an identity
source.
Note
When you install vCenter Server components with separate installers, you can choose which
account or group can log in to vCenter Server upon installation. Specify this account or group
on the Single Sign-On Information page of the installer, in the following text box: vCenter
Server administrator recognized by vCenter Single Sign-On. For example, to grant a group of
domain administrators permission to log in to vCenter Server, type of name of the domain
administrators group, such as Domain Admins@VCADSSO.LOCAL.
In high availablity and multisite Single Sign-On modes, there is no local operating system
identity source. Therefore, it will not work if you enter Administrators or Administrator
in the text box vCenter Server administrator recognized by vCenter Single Sign-On.
Administrators is treated as the local operating system group Administrators, and
Administrator is treated me as local operating system user Administrator.
2.11.8 dentity Sources for vCenter Server with vCenter Single Sign On
vCenter Server 5.1 with vCenter Single Sign On adds support for several new types of user
repository.
vCenter Server versions earlier than version 5.1 supported Active Directory and local
operating system users as user repositories. vCenter Server 5.1 supports the following types
of user repositories as identity sources.
Active Directory.
OpenLDAP.
Local operating system.
System.
vCenter Single Sign-On identity sources are managed by Single Sign-On administrator users.
You can attach multiple identity sources from each type to a single Single Sign-On server.
Each identity source has a name that is unique within the scope of the corresponding Single
Sign-On server instance. There is always exactly one System identity source, named SystemDomain.
There can be at most one local operating system identity source. On Linux systems, the
identity source label is localOS. On Windows systems, the identity source label is the
system's host name. The local operating system identity source can exist only in nonclustered Single Sign-On server deployments.
You can attach remote identity sources to a Single Sign-On server instance. Remote identity
sources are limited to any of Active Directory, and OpenLDAP server implementations.
During Single Sign On installation, the installer can automatically discover Active Directory
identity sources, if your system meets the appropriate prerequisites. See the section
Note
If you install an instance of vCenter Server as a local system account on a local SQL Server
database with Integrated Windows NT Authentication, and you add an Integrated Windows
NT Authentication user to the local database server with the same default database as
vCenter Server, vCenter Server might not start. See vCenter Server Fails to Start When
Installed as a Local System Account on a Local SQL Server Database with Integrated
Windows NT Authentication.
3 HA Clusters
3.1 Requirements for a vSphere HA Cluster
Review this list before setting up a vSphere HA cluster. For more information, follow the
appropriate cross reference or see Creating a vSphere HA Cluster.
Note
vSphere HA uses datastore heartbeating to distinguish between partitioned, isolated, and
failed hosts. Accordingly, if there are some datastores that are more reliable in your
environment, configure vSphere HA to give preference to them.
Power off all virtual machines in the cluster that are running on hosts with a feature
set greater than the EVC mode that you intend to enable, or migrate out of the
cluster.
All hosts in the cluster must meeting the following requirements.
Requirements
Description
CPUs
Advanced CPU
features enabled
Supported CPUs for To check EVC support for a specific processor or server model, see
the EVC mode that you the VMware Compatibility Guide at
want to enable
http://www.vmware.com/resources/compatibility/search.php.
Note
The default alarms include the feature name, vSphere HA.
Select the Percentage of Cluster Resources Reserved admission control policy. This
policy offers the most flexibility in terms of host and virtual machine sizing. When
configuring this policy, choose a percentage for CPU and memory that reflects the
number of host failures you want to support. For example, if you want vSphere HA to
set aside resources for two host failures and have ten hosts of equal capacity in the
cluster, then specify 20% (2/10).
Ensure that you size all cluster hosts equally. For the Host Failures Cluster Tolerates
policy, an unbalanced cluster results in excess capacity being reserved to handle
failures because vSphere HA reserves capacity for the largest hosts. For the
Percentage of Cluster Resources Policy, an unbalanced cluster requires that you
specify larger percentages than would otherwise be necessary to reserve enough
capacity for the anticipated number of host failures.
If you plan to use the Host Failures Cluster Tolerates policy, try to keep virtual
machine sizing requirements similar across all configured virtual machines. This
policy uses slot sizes to calculate the amount of capacity needed to reserve for each
virtual machine. The slot size is based on the largest reserved memory and CPU
needed for any virtual machine. When you mix virtual machines of different CPU and
memory requirements, the slot size calculation defaults to the largest possible, which
limits consolidation.
If you plan to use the Specify Failover Hosts policy, decide how many host failures to
support and then specify this number of hosts as failover hosts. If the cluster is
unbalanced, the designated failover hosts should be at least the same size as the nonfailover hosts in your cluster. This ensures that there is adequate capacity in case of
failure.
When making changes to the networks that your clustered ESXi hosts are on,
suspend the Host Monitoring feature. Changing your network hardware or
networking settings can interrupt the heartbeats that vSphere HA uses to detect host
failures, and this might result in unwanted attempts to fail over virtual machines.
When you change the networking configuration on the ESXi hosts themselves, for
example, adding port groups, or removing vSwitches, suspend Host Monitoring. After
you have made the networking configuration changes, you must reconfigure vSphere
HA on all hosts in the cluster, which causes the network information to be
reinspected. Then re-enable Host Monitoring.
Note
Because networking is a vital component of vSphere HA, if network maintenance needs to be
performed inform the vSphere HA administrator.
On legacy ESX hosts in the cluster, vSphere HA communications travel over all
networks that are designated as service console networks. VMkernel networks are not
used by these hosts for vSphere HA communications.
On ESXi hosts in the cluster, vSphere HA communications, by default, travel over
VMkernel networks, except those marked for use with vMotion. If there is only one
VMkernel network, vSphere HA shares it with vMotion, if necessary. With ESXi 4.x
and ESXi, you must also explicitly enable the Management traffic checkbox for
vSphere HA to use this network.
Note
To keep vSphere HA agent traffic on the networks you have specified, configure hosts so
vmkNICs used by vSphere HA do not share subnets with vmkNICs used for other purposes.
vSphere HA agents send packets using any pNIC that is associated with a given subnet if
there is also at least one vmkNIC configured for vSphere HA management traffic.
Consequently, to ensure network flow separation, the vmkNICs used by vSphere HA and by
other features must be on different subnets.
network partitioned. However, if the host cannot ping its isolation address, it is likely that
the host has become isolated from the network and no failover action is taken.
By default, the network isolation address is the default gateway for the host. Only one default
gateway is specified, regardless of how many management networks have been defined. You
should use the das.isolationaddress[...] advanced attribute to add isolation addresses for
additional networks. See vSphere HA Advanced Attributes.
3.3.10 Network Path Redundancy
Network path redundancy between cluster nodes is important for vSphere HA reliability. A
single management network ends up being a single point of failure and can result in failovers
although only the network has failed.
If you have only one management network, any failure between the host and the cluster can
cause an unnecessary (or false) failover activity if heartbeat datastore connectivity is not
retained during the networking failure. Possible failures include NIC failures, network cable
failures, network cable removal, and switch resets. Consider these possible sources of failure
between hosts and try to minimize them, typically by providing network redundancy.
You can implement network redundancy at the NIC level with NIC teaming, or at the
management network level. In most implementations, NIC teaming provides sufficient
redundancy, but you can use or add management network redundancy if required.
Redundant management networking allows the reliable detection of failures and prevents
isolation or partition conditions from occurring, because heartbeats can be sent over
multiple networks.
Configure the fewest possible number of hardware segments between the servers in a cluster.
The goal being to limit single points of failure. Additionally, routes with too many hops can
cause networking packet delays for heartbeats, and increase the possible points of failure.
3.3.11 Network Redundancy Using NIC Teaming
Using a team of two NICs connected to separate physical switches improves the reliability of
a management network. Because servers connected through two NICs (and through separate
switches) have two independent paths for sending and receiving heartbeats, the cluster is
more resilient. To configure a NIC team for the management network, configure the vNICs in
vSwitch configuration for Active or Standby configuration. The recommended parameter
settings for the vNICs are:
After you have added a NIC to a host in your vSphere HA cluster, you must reconfigure
vSphere HA on that host.
3.3.12 Network Redundancy Using a Second Network
As an alternative to NIC teaming for providing redundancy for heartbeats, you can create a
second management network connection, which is attached to a separate virtual switch. The
original management network connection is used for network and management purposes.
When the second management network connection is created, vSphere HA sends heartbeats
over both management network connections. If one path fails, vSphere HA still sends and
receives heartbeats over the other path.
Sphere HA Advanced Attributes
Attribute
Description
das.isolationaddress[...]
das.usedefaultisolationaddress
das.isolationshutdowntimeout
das.slotmeminmb
das.slotcpuinmhz
das.vmmemoryminmb
das.vmcpuminmhz
Description
zero. This is used for the Host Failures Cluster Tolerates
admission control policy. If no value is specified, the
default is 32MHz.
das.iostatsinterval
Select the Percentage of Cluster Resources Reserved admission control policy. This
policy offers the most flexibility in terms of host and virtual machine sizing. When
configuring this policy, choose a percentage for CPU and memory that reflects the
number of host failures you want to support. For example, if you want vSphere HA to
set aside resources for two host failures and have ten hosts of equal capacity in the
cluster, then specify 20% (2/10).
Ensure that you size all cluster hosts equally. For the Host Failures Cluster Tolerates
policy, an unbalanced cluster results in excess capacity being reserved to handle
failures because vSphere HA reserves capacity for the largest hosts. For the
Percentage of Cluster Resources Policy, an unbalanced cluster requires that you
specify larger percentages than would otherwise be necessary to reserve enough
capacity for the anticipated number of host failures.
If you plan to use the Host Failures Cluster Tolerates policy, try to keep virtual
machine sizing requirements similar across all configured virtual machines. This
policy uses slot sizes to calculate the amount of capacity needed to reserve for each
virtual machine. The slot size is based on the largest reserved memory and CPU
needed for any virtual machine. When you mix virtual machines of different CPU and
memory requirements, the slot size calculation defaults to the largest possible, which
limits consolidation.
If you plan to use the Specify Failover Hosts policy, decide how many host failures to
support and then specify this number of hosts as failover hosts. If the cluster is
unbalanced, the designated failover hosts should be at least the same size as the nonfailover hosts in your cluster. This ensures that there is adequate capacity in case of
failure.
Configuration files
protected using file
system
permissions
Detailed logging
configured.
For legacy ESX 4.x hosts, vSphere HA writes to
/var/log/vmware/fdm.
Secure vSphere HA vSphere HA logs onto the vSphere HA agents using a user account,
vpxuser, created by vCenter Server. This account is the same account
logins
used by vCenter Server to manage the host. vCenter Server creates a
random password for this account and changes the password
periodically. The time period is set by the vCenter Server
VirtualCenter.VimPasswordExpirationInDays setting. Users with
administrative privileges on the root folder of the host can log in to
the agent.
Secure
communication
Host SSL
certificate
verification
required
When making changes to the networks that your clustered ESXi hosts are on,
suspend the Host Monitoring feature. Changing your network hardware or
networking settings can interrupt the heartbeats that vSphere HA uses to detect host
failures, and this might result in unwanted attempts to fail over virtual machines.
When you change the networking configuration on the ESXi hosts themselves, for
example, adding port groups, or removing vSwitches, suspend Host Monitoring. After
you have made the networking configuration
Then re-enable Host Monitoring.
Note
Because networking is a vital component of vSphere HA, if network maintenance needs to be
performed inform the vSphere HA administrator.
On legacy ESX hosts in the cluster, vSphere HA communications travel over all
networks that are designated as service console networks. VMkernel networks are not
used by these hosts for vSphere HA communications.
On ESXi hosts in the cluster, vSphere HA communications, by default, travel over
VMkernel networks, except those marked for use with vMotion. If there is only one
VMkernel network, vSphere HA shares it with vMotion, if necessary. With ESXi 4.x
and ESXi, you must also explicitly enable the Management traffic checkbox for
vSphere HA to use this network.
Note
To keep vSphere HA agent traffic on the networks you have specified, configure hosts so
vmkNICs used by vSphere HA do not share subnets with vmkNICs used for other purposes.
vSphere HA agents send packets using any pNIC that is associated with a given subnet if
there is also at least one vmkNIC configured for vSphere HA management traffic.
Consequently, to ensure network flow separation, the vmkNICs used by vSphere HA and by
other features must be on different subnets.
heartbeats from all other hosts in the cluster. If a host can ping its network isolation address,
the host is not network isolated, and the other hosts in the cluster have either failed or are
network partitioned. However, if the host cannot ping its isolation address, it is likely that
the host has become isolated from the network and no failover action is taken.
By default, the network isolation address is the default gateway for the host. Only one default
gateway is specified, regardless of how many management networks have been defined. You
should use the das.isolationaddress[...] advanced attribute to add isolation addresses for
additional networks. See vSphere HA Advanced Attributes.
3.4.4 Network Path Redundancy
Network path redundancy between cluster nodes is important for vSphere HA reliability. A
single management network ends up being a single point of failure and can result in failovers
although only the network has failed.
If you have only one management network, any failure between the host and the cluster can
cause an unnecessary (or false) failover activity if heartbeat datastore connectivity is not
retained during the networking failure. Possible failures include NIC failures, network cable
failures, network cable removal, and switch resets. Consider these possible sources of failure
between hosts and try to minimize them, typically by providing network redundancy.
You can implement network redundancy at the NIC level with NIC teaming, or at the
management network level. In most implementations, NIC teaming provides sufficient
redundancy, but you can use or add management network redundancy if required.
Redundant management networking allows the reliable detection of failures and prevents
isolation or partition conditions from occurring, because heartbeats can be sent over
multiple networks.
Configure the fewest possible number of hardware segments between the servers in a cluster.
The goal being to limit single points of failure. Additionally, routes with too many hops can
cause networking packet delays for heartbeats, and increase the possible points of failure.
3.4.5 Network Redundancy Using NIC Teaming
Using a team of two NICs connected to separate physical switches improves the reliability of
a management network. Because servers connected through two NICs (and through separate
switches) have two independent paths for sending and receiving heartbeats, the cluster is
more resilient. To configure a NIC team for the management network, configure the vNICs in
vSwitch configuration for Active or Standby configuration. The recommended parameter
settings for the vNICs are:
After you have added a NIC to a host in your vSphere HA cluster, you must reconfigure
vSphere HA on that host.
3.4.6 Network Redundancy Using a Second Network
As an alternative to NIC teaming for providing redundancy for heartbeats, you can create a
second management network connection, which is attached to a separate virtual switch. The
original management network connection is used for network and management purposes.
When the second management network connection is created, vSphere HA sends heartbeats
over both management network connections. If one path fails, vSphere HA still sends and
receives heartbeats over the other path.
Note
The failover of fault tolerant virtual machines is independent of vCenter Server, but you must
use vCenter Server to set up your Fault Tolerance clusters.
Note
For legacy hosts prior to ESX/ESXi 4.1, this tab lists the host build number instead. Patches
can cause host build numbers to vary between ESX and ESXi installations. To ensure that
your legacy hosts are FT compatible, do not mix legacy ESX and ESXi hosts in an FT pair.
ESXi hosts have access to the same virtual machine datastores and networks. See
Best Practices for Fault Tolerance.
Fault Tolerance logging and VMotion networking configured. See Configure
Networking for Host Machines in the vSphere Client or Configure Networking for
Host Machines in the vSphere Web Client.
vSphere HA cluster created and enabled. See Creating a vSphere HA Cluster. vSphere
HA must be enabled before you can power on fault tolerant virtual machines or add a
host to a cluster that already supports fault tolerant virtual machines.
Hosts must have processors from the FT-compatible processor group. It is also highly
recommended that the hosts' processors are compatible with one another. See the
VMware knowledge base article at http://kb.vmware.com/kb/1008027 for
information on supported processors.
Hosts must be licensed for Fault Tolerance.
Hosts must be certified for Fault Tolerance. See
http://www.vmware.com/resources/compatibility/search.php and select Search by
Fault Tolerant Compatible Sets to determine if your hosts are certified.
The configuration for each host must have Hardware Virtualization (HV) enabled in
the BIOS.
To confirm the compatibility of the hosts in the cluster to support Fault Tolerance, you can
also run profile compliance checks as described in Create Cluster and Check Compliance in
the vSphere Client or Create Cluster and Check Compliance in the vSphere Web Client.
3.5.4 Virtual Machine Requirements for Fault Tolerance
You must meet the following virtual machine requirements before you use Fault Tolerance.
Before configuring vSphere Fault Tolerance, you should be aware of the features and
products Fault Tolerance cannot interoperate with.
3.5.5.1 vSphere Features Not Supported with Fault Tolerance
The following vSphere features are not supported for fault tolerant virtual machines.
Linked clones. You cannot enable Fault Tolerance on a virtual machine that is a
linked clone, nor can you create a linked clone from an FT-enabled virtual machine.
Virtual Machine Backups. You cannot back up an FT-enabled virtual machine using
Storage API for Data Protection, vSphere Data Protection, or similar backup products
that require the use of a virtual machine snapshot, as performed by ESXi. To back up
a fault tolerant virtual machine in this manner, you must first disable FT, then reenable FT after performing the backup. Storage array-based snapshots do not affect
FT.
Corrective Action
Symmetric multiprocessor
(SMP) virtual machines. Only Reconfigure the virtual machine as a single vCPU. Many
virtual machines with a single workloads have good performance configured as a single
vCPU are compatible with
vCPU.
Fault Tolerance.
Physical Raw Disk mapping
(RDM).
N_Port ID Virtualization
(NPIV).
NIC passthrough.
Virtual disks backed with thinprovisioned storage or thick- When you turn on Fault Tolerance, the conversion to the
provisioned disks that do not appropriate disk format is performed by default. You must
have clustering features
power off the virtual machine to trigger this conversion.
enabled.
Features and Devices Incompatible with Fault Tolerance and Corrective Actions
Incompatible Feature or
Device
Corrective Action
The hot plug feature is automatically disabled for fault
tolerant virtual machines. To hot plug devices (either adding
or removing), you must momentarily turn off Fault
Tolerance, perform the hot plug, and then turn on Fault
Tolerance.
Note
Hot-plugging devices.
Hosts running the Primary and Secondary VMs should operate at approximately the
same processor frequencies, otherwise the Secondary VM might be restarted more
frequently. Platform power management features that do not adjust based on
workload (for example, power capping and enforced low frequency modes to save
power) can cause processor frequencies to vary greatly. If Secondary VMs are being
restarted on a regular basis, disable all power management modes on the hosts
running fault tolerant virtual machines or ensure that all hosts are running in the
same power management modes.
Apply the same instruction set extension configuration (enabled or disabled) to all
hosts. The process for enabling or disabling instruction sets varies among BIOSes.
See the documentation for your hosts' BIOSes about how to configure instruction
sets.
When errors related to your implementation of Fault Tolerance are generated by vCenter
Server, the Fault Details screen appears.
This screen lists faults related to Fault Tolerance and for each fault it provides the type of
fault (red is an error, yellow is a warning), the name of the virtual machine or host involved,
and a brief description.
You can also invoke this screen for a specific failed Fault Tolerance task. To do this, select the
task in either the Recent Tasks pane or the Tasks & Events tab for the entity that experienced
the fault and click the View details link that appears in the Details column.
3.5.6.7 Viewing Fault Tolerance Errors in the vSphere Web Client
When tasks related to your implementation of Fault Tolerance cause errors, you can view
information about them in the Recent Tasks pane.
The Recent Tasks pane lists a summary of each error under the Failed tab. For information
about failed tasks, click More Tasks to open the Task Console.
In the Task Console, each task is listed with information that includes its Name, Target, and
Status. In the Status column, if the task failed, the type of fault it generated is described. For
information about a task, select it and details appear in the pane below the task list.
3.5.6.8 Upgrade Hosts Used for Fault Tolerance
When you upgrade hosts that contain fault tolerant virtual machines, ensure that the
Primary and Secondary VMs continue to run on hosts with the same FT version number or
host build number (for hosts prior to ESX/ESXi 4.1).
3.5.6.9 Prerequisites
Verify that you have cluster administrator privileges.
Verify that you have sets of four or more ESXi hosts that are hosting fault tolerant virtual
machines that are powered on. If the virtual machines are powered off, the Primary and
Secondary VMs can be relocated to hosts with different builds.
This upgrade procedure is for a minimum four-node cluster. The same instructions can be
followed for a smaller cluster, though the unprotected interval will be slightly longer.
Procedure
Using vMotion, migrate the fault tolerant virtual machines off of two hosts.
Upgrade the two evacuated hosts to the same ESXi build.
Turn off Fault Tolerance on the Primary VM.
Using vMotion, move the disabled Primary VM to one of the upgraded hosts.
Turn on Fault Tolerance on the Primary VM that was moved.
Repeat Step 1 to Step 5 for as many fault tolerant virtual machine pairs as can be
accommodated on the upgraded hosts.
In addition to non-fault tolerant virtual machines, you should have no more than four
fault tolerant virtual machines (primaries or secondaries) on any single host. The
number of fault tolerant virtual machines that you can safely run on each host is
based on the sizes and workloads of the ESXi host and virtual machines, all of which
can vary.
If you are using NFS to access shared storage, use dedicated NAS hardware with at
least a 1Gbit NIC to obtain the network performance required for Fault Tolerance to
work properly.
Ensure that a resource pool containing fault tolerant virtual machines has excess
memory above the memory size of the virtual machines. The memory reservation of a
fault tolerant virtual machine is set to the virtual machine's memory size when Fault
Tolerance is turned on. Without this excess in the resource pool, there might not be
any memory available to use as overhead memory.
Use a maximum of 16 virtual disks per fault tolerant virtual machine.
To ensure redundancy and maximum Fault Tolerance protection, you should have a
minimum of three hosts in the cluster. In a failover situation, this provides a host that
can accommodate the new Secondary VM that is created.
4 Networking
4.1 vSphere Distributed Switch Health Check
vSphere 5.1 distributed switch health check helps identify and troubleshoot configuration
errors in vSphere distributed switches.
The following errors are common configuration errors that health check helps identify.
Mismatched VLAN trunks between a vSphere distributed switch and physical switch.
Mismatched MTU settings between physical network adapters, distributed switches,
and physical switch ports.
Mismatched virtual switch teaming policies for the physical switch port-channel
settings.
VLAN. Checks whether vSphere distributed switch VLAN settings match trunk port
configuration on the adjacent physical switch ports.
MTU. Checks whether the physical access switch port MTU jumbo frame setting
based on per VLAN matches the vSphere distributed switch MTU setting.
Teaming policies. Checks whether the physical access switch ports EtherChannel
setting matches the distributed switch distributed port group IPHash teaming policy
settings.
Health check is limited to only the access switch port to which the distributed switch uplink
connects.
Note
For VLAN and MTU checks, you must have at least two link-up physical uplink NICs for the
distributed switch.
For a teaming policy check, you must have at least two link-up physical uplink NICs and two
hosts when applying the policy.
Description
Port binding Choose when ports are assigned to virtual machines connected to this
distributed port group.
Static binding: Assign a port to a virtual machine when the virtual machine
connects to the distributed port group. This option is not available when the
vSphere Web Client is connected directly to ESXi.
Dynamic binding: Assign a port to a virtual machine the first time the
virtual machine powers on after it is connected to the distributed port
group. Dynamic binding is deprecated in ESXi 5.0.
Ephemeral: No port binding. This option is not available when the vSphere
Web Client is connected directly to ESXi.
Port
allocation
Elastic: The default number of ports is eight. When all ports are assigned, a
new set of eight ports is created. This is the default.
Fixed: The default number of ports is set to eight. No additional ports are
created when all ports are assigned.
Number of
ports
Network
Use the drop-down menu to assign the new distributed port group to a userresource pool defined network resource pool. If you have not created a network resource
pool, this menu is empty.
VLAN
Advanced
Select this check box to customize the policy configurations for the new
distributed port group.
(Optional) In the Security section, edit the security exceptions and click Next.
Setting
Description
Promiscuous Reject. Placing a guest adapter in promiscuous mode has no effect on which
mode
frames are received by the adapter.
Accept. Placing a guest adapter in promiscuous mode causes it to detect all
frames passed on the vSphere distributed switch. These frames are allowed
under the VLAN policy for the port group to which the adapter is connected.
MAC address Reject. If you set to Reject and the guest operating system changes the MAC
address of the adapter to anything other than what is in the .vmx configuration
changes
file, all inbound frames are dropped.
If the Guest OS changes the MAC address back to match the MAC address in
the .vmx configuration file, inbound frames are passed again.
Accept. Changing the MAC address from the Guest OS has the intended effect:
Reject. Any outbound frame with a source MAC address that is different from
the one currently set on the adapter is dropped.
Accept. No filtering is performed and all outbound frames are passed.
Setting
Description
Status
If you enable either Ingress Traffic Shaping or Egress Traffic Shaping, you are
setting limits on the amount of networking bandwidth allocated for each virtual
adapter associated with this particular port group. If you disable the policy,
services have a free, clear connection to the physical network by default.
Average
Bandwidth
Establishes the number of bits per second to allow across a port, averaged over
time. This is the allowed average load.
Peak
Bandwidth
The maximum number of bits per second to allow across a port when it is
sending and receiving a burst of traffic. This tops the bandwidth used by a port
whenever it is using its burst bonus.
Burst Size
(Optional) In the Teaming and failover section, edit the settings and click Next.
Setting
Description
Load
balancing
Note
IP-based teaming requires that the physical switch be configured with
etherchannel. For all other options, disable etherchannel.
Network
failover
detection
Link Status only. Relies solely on the link status that the network
adapter provides. This option detects failures, such as cable pulls and
physical switch power failures, but not configuration errors, such as a
physical switch port being blocked by spanning tree or that is
misconfigured to the wrong VLAN or cable pulls on the other side of a
physical switch.
Beacon Probing. Sends out and listens for beacon probes on all NICs
in the team and uses this information, in addition to link status, to
determine link failure. This detects many of the failures previously
mentioned that are not detected by link status alone.
Note
Do not use beacon probing with IP-hash load balancing.
Notify
switches
Select Yes or No to notify switches in the case of failover. If you select Yes,
whenever a virtual NIC is connected to the distributed switch or whenever
that virtual NICs traffic would be routed over a different physical NIC in the
team because of a failover event, a notification is sent out over the network to
update the lookup tables on physical switches. In almost all cases, this
process is desirable for the lowest latency of failover occurrences and
migrations with vMotion.
traffic resource pools each receive 25% of the available bandwidth. The remaining resource
pools each receive 12.5% of the available bandwidth. These reservations apply only when the
physical adapter is saturated.
Note
The iSCSI traffic resource pool shares do not apply to iSCSI traffic on a dependent hardware
iSCSI adapter.
The host limit of a network resource pool is the upper limit of bandwidth that the network
resource pool can use.
Assigning a QoS priority tag to a network resource pool applies an 802.1p tag to all outgoing
packets associated with that network resource pool.
Select the Physical adapter shares for the network resource pool.
Option
Custom
Description
Type a specific number of shares, from 1 to 100, for this network
resource pool.
High
Normal
Low
Microsoft Windows 2003 Enterprise Edition with Service Pack 2 (32 bit and 64 bit)
Red Hat Enterprise Linux 4 (64 bit)
Red Hat Enterprise Linux 5 (32 bit and 64 bit)
Requirements
Hosts with Intel processors require ESXi 5.1 or later.
Hosts with AMD processors are not supported with SR-IOV.
Must be compatible with the ESXi release.
Must have an Intel processor.
Must not have an AMD processor.
Physical host
Requirements
Must be compatible with the ESXi release.
Physical NIC
Must be supported for use with the host and SR-IOV according to the
technical documentation from the server vendor.
Must have SR-IOV enabled in the firmware.
Must be certified by VMware.
PF driver in ESXi for Must be installed on the ESXi host. The ESXi release provides a
the physical NIC
default driver for certain NICs, while for others you must download
and manually install it.
Guest OS
o verify compatibility of physical hosts and NICs with ESXi releases, see the VMware
Compatibility Guide.
4.5.1.2 Availability of Features
The following features are not available for virtual machines configured with SR-IOV:
vMotion
Storage vMotion
vShield
Netflow
Virtual Wire
High Availability
Fault Tolerance
DRS
DPM
Suspend and resume
Snapshots
MAC-based VLAN for passthrough virtual functions
Hot addition and removal of virtual devices, memory, and vCPU
Participation in a cluster environment
Note
Attempts to enable or configure unsupported features with SR-IOV in the vSphere Web
Client result in unexpected behavior in your environment.
Products based on the Intel 82599ES 10 Gigabit Ethernet Controller Family (Niantic)
Products based on the Intel Ethernet Controller X540 Family (Twinville)
Emulex OneConnect (BE3)
If you upgrade from vSphere 5.0 or earlier to vSphere 5.1 or later, SR-IOV support is not
available until you update the NIC drivers for the vSphere release. NICs must have firmware
and drivers that support SR-IOV enabled for SR-IOV functionality to operate.
4.5.2 vSphere 5.1 and Virtual Function Interaction
Virtual functions (VFs) are lightweight PCIe functions that contain all the resources
necessary for data movement but have a carefully minimized set of configuration resources.
There are some restrictions in the interactions between vSphere 5.1 and VFs.
When a physical NIC creates VFs for SR-IOV to use, the physical NIC becomes a
hidden uplink and cannot be used as a normal uplink. This means it cannot be added
to a standard or distributed switch.
There is no rate control for VFs in vSphere 5.1. Every VF could potentially use the
entire bandwidth for a physical link.
When a VF device is configured as a passthrough device on a virtual machine, the
standby and hibernate functions for the virtual machine are not supported.
Due to the limited number of vectors available for passthrough devices, there is a
limited number of VFs supported on an vSphere ESXi host . vSphere 5.1 SR-IOV
supports up to 41 VFs on supported Intel NICs and up to 64 VFs on supported
Emulex NICs.
4.5.3.1 Configure SR-IOV in a Host Profile with the vSphere Web Client
Before you can connect a virtual machine to a virtual function, you have to configure the
virtual functions of the physical NIC on your host by using a host profile.
You can enable SR-IOV virtual functions on the host by using the esxcli system module
parameters set vCLI command on the NIC driver parameter for virtual functions in
accordance with the driver documentation. For more information about using vCLI
commands, see vSphere Command-Line Interface Documentation.
4.5.3.2 LACP Limitations on a vSphere Distributed Switch
Link Aggregation Control Protocol (LACP) on a vSphere distributed switch allows network
devices to negotiate automatic bundling of links by sending LACP packets to a peer.
However, there are some limitations when using LACP with a vSphere distributed switch.
LACP only works with IP Hash load balancing and Link Status Network failover
detection.
LACP is not compatible with iSCSI software multipathing.
vSphere only supports one LACP group per distributed switch, and only one LACP
group per host.
LACP settings do not exist in host profiles.
LACP between two nested ESXi hosts is not possible.
LACP does not work with port mirroring.
Option
Description
Promiscuous
Mode
MAC Address
Changes
Reject If you set the MAC Address Changes to Reject and the guest
operating system changes the MAC address of the adapter to anything
other than what is in the .vmx configuration file, all inbound frames are
dropped.
If the Guest OS changes the MAC address back to match the MAC address
in the .vmx configuration file, inbound frames are passed again.
Accept Changing the MAC address from the Guest OS has the intended
effect: frames to the new MAC address are received.
Forged
Transmits
Reject Any outbound frame with a source MAC address that is different
from the one currently set on the adapter are dropped.
Accept No filtering is performed and all outbound frames are passed.
Log in to the vSphere Client and select the Networking inventory view.
Right-click the vSphere distributed switch in the inventory pane, and select Edit
Settings.
Navigate to the NetFlow tab.
Type the IP address and Port of the NetFlow collector.
Type the VDS IP address.
With an IP address to the vSphere distributed switch, the NetFlow collector can
interact with the vSphere distributed switch as a single switch, rather than interacting
with a separate, unrelated switch for each associated host.
(Optional) Use the up and down menu arrows to set the Active flow export timeout
and Idle flow export timeout.
(Optional) Use the up and down menu arrows to set the Sampling rate.
The sampling rate determines what portion of data NetFlow collects, with the
sampling rate number determining how often NetFlow collects the packets. A
collector with a sampling rate of 2 collects data from every other packet. A collector
with a sampling rate of 5 collects data from every fifth packet.
(Optional) Select Process internal flows only to collect data only on network activity
between virtual machines on the same host.
Click OK.
4.6.1 CDP
Option
Description
Listen
ESXi detects and displays information about the associated Cisco switch port, but
information about the vSphere distributed switch is not available to the Cisco
switch administrator.
AdvertiseESXi makes information about the vSphere distributed switch available to the
Cisco switch administrator, but does not detect and display information about the
Cisco switch.
Both
ESXi detects and displays information about the associated Cisco switch and
makes information about the vSphere distributed switch available to the Cisco
switch administrator.
ESXi supports VMkernel-based NFS mounts for storing virtual disks on NFS datastores.
In addition to storing virtual disks on NFS datastores, you can also use NFS Datastores as a
central repository for ISO images and virtual machine templates. For more information
about creating NFS datastores, see vSphere Storage.
ESXi supports NFS version 3 over Layer 2 and Layer 3 Network switches. Host servers and
NFS storage arrays must be on different subnets and the network switch must handle the
routing information.
Separate network services from one another to achieve greater security and better
performance.
Put a set of virtual machines on a separate physical NIC. This separation allows for a
portion of the total networking workload to be shared evenly across multiple CPUs.
The isolated virtual machines can then better serve traffic from a Web client, for
example
Keep the vMotion connection on a separate network devoted to vMotion. When
migration with vMotion occurs, the contents of the guest operating systems memory
is transmitted over the network. You can do this either by using VLANs to segment a
single physical network or by using separate physical networks (the latter is
preferable).
When using passthrough devices with a Linux kernel version 2.6.20 or earlier, avoid
MSI and MSI-X modes because these modes have significant performance impact.
To physically separate network services and to dedicate a particular set of NICs to a
specific network service, create a vSphere standard switch or vSphere distributed
switch for each service. If this is not possible, separate network services on a single
switch by attaching them to port groups with different VLAN IDs. In either case,
confirm with your network administrator that the networks or VLANs you choose are
isolated in the rest of your environment and that no routers connect them.
You can add and remove network adapters from a standard or distributed switch
without affecting the virtual machines or the network service that is running behind
that switch. If you remove all the running hardware, the virtual machines can still
communicate among themselves. If you leave one network adapter intact, all the
virtual machines can still connect with the physical network.
To protect your most sensitive virtual machines, deploy firewalls in virtual machines
that route between virtual networks with uplinks to physical networks and pure
virtual networks with no uplinks.
For best performance, use vmxnet3 virtual NICs.
Every physical network adapter connected to the same vSphere standard switch or
vSphere distributed switch should also be connected to the same physical network.
Configure all VMkernel network adapters to the same MTU. When several VMkernel
network adapters are connected to vSphere distributed switches but have different
MTUs configured, you might experience network connectivity problems.
When creating a distributed port group, do not use dynamic binding. Dynamic
binding is deprecated in ESXi 5.0.
5 Storage
5.1 Making LUN Decisions
You must plan how to set up storage for your ESXi systems before you format LUNs with
VMFS datastores.
When you make your LUN decision, keep in mind the following considerations:
Each LUN should have the correct RAID level and storage characteristic for the
applications running in virtual machines that use the LUN.
Each LUN must contain only one VMFS datastore.
If multiple virtual machines access the same VMFS, use disk shares to prioritize
virtual machines.
You might want fewer, larger LUNs for the following reasons:
More flexibility to create virtual machines without asking the storage administrator
for more space.
More flexibility for resizing virtual disks, doing snapshots, and so on.
Fewer VMFS datastores to manage.
You might want more, smaller LUNs for the following reasons:
When the storage characterization for a virtual machine is not available, there is often no
simple method to determine the number and size of LUNs to provision. You can experiment
using either a predictive or adaptive scheme.
5.1.1 Use the Predictive Scheme to Make LUN Decisions
When setting up storage for ESXi systems, before creating VMFS datastores, you must
decide on the size and number of LUNs to provision. You can experiment using the
predictive scheme.
Procedure
Note
Disk shares are relevant only within a given host. The shares assigned to virtual machines on
one host have no effect on virtual machines on other hosts.
Provision a large LUN (RAID 1+0 or RAID 5), with write caching enabled.
Create a VMFS on that LUN.
Create four or five virtual disks on the VMFS.
Run the applications to determine whether disk performance is acceptable.
If performance is acceptable, you can place additional virtual disks on the VMFS. If
performance is not acceptable, create a new, large LUN, possibly with a different RAID level,
and repeat the process. Use migration so that you do not lose virtual machines data when
you recreate the LUN.
5.1.3 NPIV Capabilities and Limitations
Learn about specific capabilities and limitations of the use of NPIV with ESXi.
ESXi with NPIV supports the following items:
NPIV supports vMotion. When you use vMotion to migrate a virtual machine it
retains the assigned WWN.
If you migrate an NPIV-enabled virtual machine to a host that does not support
NPIV, VMkernel reverts to using a physical HBA to route the I/O.
If your FC SAN environment supports concurrent I/O on the disks from an activeactive array, the concurrent I/O to two different NPIV ports is also supported.
When you use ESXi with NPIV, the following limitations apply:
ESXi system
requirements
Adapter
requirements
Enable and correctly configure the adapter, so it can access the boot LUN.
See your vendor documentation.
Access control
Each host must have access to its own boot LUN only, not the boot LUNs
of other hosts. Use storage system software to make sure that the host
accesses only the designated LUNs.
Multiple servers can share a diagnostic partition. You can use array
specific LUN masking to achieve this.
Multipathing
support
SAN
considerations
Hardwarespecific
considerations
If you are running an IBM eServer BladeCenter and use boot from SAN,
you must disable IDE drives on the blades.
If there are no running virtual machines on the VMFS datastore, after you change the ID of
the LUN, you must use rescan to reset the ID on your host. For information on using rescan,
see Storage Refresh and Rescan Operations.
storage arrays. Make sure that the paths through the switch fabric are not saturated, that is,
that the switch fabric is running at the highest throughput.
5.5.1 Storage Array Performance
Storage array performance is one of the major factors contributing to the performance of the
entire SAN environment.
If there are issues with storage array performance, be sure to consult your storage array
vendors documentation for any relevant information.
Follow these general guidelines to improve the array performance in the vSphere
environment:
When assigning LUNs, remember that each LUN is accessed by a number of hosts,
and that a number of virtual machines can run on each host. One LUN used by a host
can service I/O from many different applications running on different operating
systems. Because of this diverse workload, the RAID group containing the ESXi
LUNs should not include LUNs used by other servers that are not running ESXi.
Make sure read/write caching is enabled.
SAN storage arrays require continual redesign and tuning to ensure that I/O is load
balanced across all storage array paths. To meet this requirement, distribute the
paths to the LUNs among all the SPs to provide optimal load balancing. Close
monitoring indicates when it is necessary to rebalance the LUN distribution.
Tuning statically balanced storage arrays is a matter of monitoring the specific performance
statistics (such as I/O operations per second, blocks per second, and response time) and
distributing the LUN workload to spread the workload across all the SPs.
Note
Dynamic load balancing is not currently supported with ESXi.
Because each application has different requirements, you can meet these goals by choosing
an appropriate RAID group on the storage array. To achieve performance goals:
Place each LUN on a RAID group that provides the necessary performance levels. Pay
attention to the activities and resource utilization of other LUNS in the assigned
RAID group. A high-performance RAID group that has too many applications doing
I/O to it might not meet performance goals required by an application running on the
ESXi host.
Make sure that each server has a sufficient number of HBAs to allow maximum
throughput for all the applications hosted on the server for the peak period. I/O
spread across multiple HBAs provide higher throughput and less latency for each
application.
To provide redundancy in the event of HBA failure, make sure the server is connected
to a dual redundant fabric.
When allocating LUNs or RAID groups for ESXi systems, multiple operating systems
use and share that resource. As a result, the performance required from each LUN in
the storage subsystem can be much higher if you are working with ESXi systems than
if you are using physical machines. For example, if you expect to run four I/O
intensive applications, allocate four times the performance capacity for the ESXi
LUNs.
When using multiple ESXi systems in conjunction with vCenter Server, the
performance needed from the storage subsystem increases correspondingly.
The number of outstanding I/Os needed by applications running on an ESXi system
should match the number of I/Os the HBA and storage array can handle.
Comments
Topology
EMC Symmetrix
Enable the SPC2 and SC3 settings. Contact EMC for the latest
settings.
EMC Clariion
Comments
details.
Host type must be Linux.
HP MSA
5.6 iSCSI
5.6.1 iSCSI Naming Conventions
iSCSI uses a special unique name to identify an iSCSI node, either target or initiator. This
name is similar to the WorldWide Name (WWN) associated with Fibre Channel devices and
is used as a way to universally identify the node.
iSCSI names are formatted in two different ways. The most common is the IQN format.
For more details on iSCSI naming requirements and string profiles, see RFC 3721 and RFC
3722 on the IETF Web site.
5.6.1.1 iSCSI Qualified Name (IQN) Format
The IQN format takes the form iqn.yyyy-mm.naming-authority:unique name, where:
yyyy-mm is the year and month when the naming authority was established.
naming-authority is usually reverse syntax of the Internet domain name of the
naming authority. For example, the iscsi.vmware.com naming authority could have
the iSCSI qualified name form of iqn.1998-01.com.vmware.iscsi. The name indicates
that the vmware.com domain name was registered in January of 1998, and iscsi is a
subdomain, maintained by vmware.com.
unique name is any name you want to use, for example, the name of your host. The
naming authority must make sure that any names assigned following the colon are
unique, such as:
o iqn.1998-01.com.vmware.iscsi:name1
o iqn.1998-01.com.vmware.iscsi:name2
o iqn.1998-01.com.vmware.iscsi:name999
The types of storage that your host supports include active-active, active-passive, and ALUAcompliant.
Active-active
storage system
Allows access to the LUNs simultaneously through all the storage ports
that are available without significant performance degradation. All the
paths are active at all times, unless a path fails.
Active-passive
storage system
Asymmetrical
storage system
Virtual port
storage system
Allows access to all available LUNs through a single virtual port. These
are active-active storage devices, but hide their multiple connections
though a single port. ESXi multipathing does not make multiple
connections from a specific port to the storage by default. Some storage
vendors supply session managers to establish and manage multiple
connections to their storage. These storage systems handle port failover
and connection balancing transparently. This is often referred to as
transparent failover.
When you use any dependent hardware iSCSI adapter, performance reporting for a
NIC associated with the adapter might show little or no activity, even when iSCSI
traffic is heavy. This behavior occurs because the iSCSI traffic bypasses the regular
networking stack.
If you use a third-party virtual switch, for example Cisco Nexus 1000V DVS, disable
automatic pinning. Use manual pinning instead, making sure to connect a VMkernel
adapter (vmk) to an appropriate physical NIC (vmnic). For information, refer to your
virtual switch vendor documentation.
The Broadcom iSCSI adapter performs data reassembly in hardware, which has a
limited buffer space. When you use the Broadcom iSCSI adapter in a congested
network or under heavy load, enable flow control to avoid performance degradation.
Flow control manages the rate of data transmission between two nodes to prevent a
fast sender from overrunning a slow receiver. For best results, enable flow control at
the end points of the I/O path, at the hosts and iSCSI storage systems.
To enable flow control for the host, use the esxcli system module parameters
command. For details, see the VMware knowledge base article at
http://kb.vmware.com/kb/1013413
Make sure that the VMkernel network adapters are assigned addresses on the same
subnet as the iSCSI storage portal they connect to.
iSCSI adapters using VMkernel adapters are not able to connect to iSCSI ports on
different subnets, even if those ports are discovered by the iSCSI adapters.
When using separate vSphere switches to connect physical network adapters and
VMkernel adapters, make sure that the vSphere switches connect to different IP
subnets.
If VMkernel adapters are on the same subnet, they must connect to a single vSwitch.
If you migrate VMkernel adapters to a different vSphere switch, move associated
physical adapters.
Do not make configuration changes to iSCSI-bound VMkernel adapters or physical
network adapters.
Do not make changes that might break association of VMkernel adapters and
physical network adapters. You can break the association if you remove one of the
adapters or the vSphere switch that connects them, or change the 1:1 network policy
for their connection.
Description
Supported
Software iSCSI
None
Software iSCSI
Dependent
hardware iSCSI
Software iSCSI
Use unidirectional CHAP The host prefers CHAP, but can use nonunless prohibited by
CHAP connections if the target does not
target
support CHAP.
Dependent
hardware iSCSI
Independent
hardware iSCSI
Software iSCSI
Software iSCSI
The host and the target support bidirectional
Dependent
CHAP.
hardware iSCSI
ESXi hosts can boot from an iSCSI SAN using the software or dependent hardware iSCSI
adapters and network adapters.
To deploy ESXi and boot from the iSCSI SAN, the host must have an iSCSI boot capable
network adapter that supports the iSCSI Boot Firmware Table (iBFT) format. The iBFT is a
method of communicating parameters about the iSCSI boot device to an operating system.
Before installing ESXi and booting from the iSCSI SAN, configure the networking and iSCSI
boot parameters on the network adapter and enable the adapter for the iSCSI boot. Because
configuring the network adapter is vendor specific, review your vendor documentation for
instructions.
When you first boot from iSCSI, the iSCSI boot firmware on your system connects to an
iSCSI target. If login is successful, the firmware saves the networking and iSCSI boot
parameters in the iBFT and stores the table in the system's memory. The system uses this
table to configure its own iSCSI connection and networking and to start up.
The following list describes the iBFT iSCSI boot sequence.
When restarted, the system BIOS detects the iSCSI boot firmware on the network
adapter.
The iSCSI boot firmware uses the preconfigured boot parameters to connect with the
specified iSCSI target.
If the connection to the iSCSI target is successful, the iSCSI boot firmware writes the
networking and iSCSI boot parameters in to the iBFT and stores
Note
The system uses this table to configure its own iSCSI connection and networking and
to start up.
4.
5.
6.
7.
IPv6
Note
Update your NIC's boot code and iBFT firmware using vendor supplied tools before trying to
install and boot VMware ESXi. Consult vendor documentation and VMware HCL for
supported boot code and iBFT firmware versions for VMware ESXi iBFT boot. The boot code
and iBFT firmware released by vendors prior to the ESXi 4.1 release might not work.
After you set up your host to boot from iBFT iSCSI, the following restrictions apply:
You cannot disable the software iSCSI adapter. If the iBFT configuration is present in
the BIOS, the host re-enables the software iSCSI adapter during each reboot.
Note
If you do not use the iBFT-enabled network adapter for the iSCSI boot and do not want the
software iSCSI adapter to be always enabled, remove the iBFT configuration from the
network adapter.
You cannot remove the iBFT iSCSI boot target using the vSphere Client or the
vSphere Web Client. The target appears on the list of adapter static targets.
You should observe these tips for avoiding problems with your SAN configuration:
Place only one VMFS datastore on each LUN. Multiple VMFS datastores on one LUN
is not recommended.
Do not change the path policy the system sets for you unless you understand the
implications of making such a change.
Document everything. Include information about configuration, access control,
storage, switch, server and iSCSI HBA configuration, software and firmware versions,
and storage cable plan.
Make several copies of your topology maps. For each element, consider what happens
to your SAN if the element fails.
Cross off different links, switches, HBAs and other elements to ensure you did not
miss a critical failure point in your design.
Ensure that the iSCSI HBAs are installed in the correct slots in the ESXi host, based
on slot and bus speed. Balance PCI bus load among the available busses in the server.
Become familiar with the various monitor points in your storage network, at all
visibility points, including ESXi performance charts, Ethernet switch statistics, and
storage performance statistics.
Be cautious when changing IDs of the LUNs that have VMFS datastores being used
by your host. If you change the ID, virtual machines running on the VMFS datastore
will fail.
If there are no running virtual machines on the VMFS datastore, after you change the
ID of the LUN, you must use rescan to reset the ID on your host. For information on
using rescan, see Storage Refresh and Rescan Operations.
If you need to change the default iSCSI name of your iSCSI adapter, make sure the
name you enter is worldwide unique and properly formatted. To avoid storage access
problems, never assign the same iSCSI name to different adapters, even on different
hosts.
systems. Because of this diverse workload, the RAID group that contains the ESXi LUNs
should not include LUNs that other hosts use that are not running ESXi for I/O intensive
applications.
Enable read caching and write caching.
Load balancing is the process of spreading server I/O requests across all available SPs and
their associated host server paths. The goal is to optimize performance in terms of
throughput (I/O per second, megabytes per second, or response times).
SAN storage systems require continual redesign and tuning to ensure that I/O is load
balanced across all storage system paths. To meet this requirement, distribute the paths to
the LUNs among all the SPs to provide optimal load balancing. Close monitoring indicates
when it is necessary to manually rebalance the LUN distribution.
Tuning statically balanced storage systems is a matter of monitoring the specific
performance statistics (such as I/O operations per second, blocks per second, and response
time) and distributing the LUN workload to spread the workload across all the SPs.
5.11.2 Server Performance with iSCSI
You must consider several factors to ensure optimal server performance.
Each server application must have access to its designated storage with the following
conditions:
Because each application has different requirements, you can meet these goals by choosing
an appropriate RAID group on the storage system. To achieve performance goals, perform
the following tasks:
Place each LUN on a RAID group that provides the necessary performance levels. Pay
attention to the activities and resource utilization of other LUNS in the assigned
RAID group. A high-performance RAID group that has too many applications doing
I/O to it might not meet performance goals required by an application running on the
ESXi host.
Provide each server with a sufficient number of network adapters or iSCSI hardware
adapters to allow maximum throughput for all the applications hosted on the server
for the peak period. I/O spread across multiple ports provides higher throughput and
less latency for each application.
To provide redundancy for software iSCSI, make sure the initiator is connected to all
network adapters used for iSCSI connectivity.
When allocating LUNs or RAID groups for ESXi systems, multiple operating systems
use and share that resource. As a result, the performance required from each LUN in
the storage subsystem can be much higher if you are working with ESXi systems than
if you are using physical machines. For example, if you expect to run four I/O
intensive applications, allocate four times the performance capacity for the ESXi
LUNs.
When using multiple ESXi systems in conjunction with vCenter Server, the
performance needed from the storage subsystem increases correspondingly.
The number of outstanding I/Os needed by applications running on an ESXi system
should match the number of I/Os the SAN can handle.
When systems read data from storage, the maximum response from the storage is to send
enough data to fill the link between the storage systems and the Ethernet switch. It is
unlikely that any single system or virtual machine gets full use of the network speed, but this
situation can be expected when many systems share one storage device.
When writing data to storage, multiple systems or virtual machines might attempt to fill
their links. As Dropped Packets shows, when this happens, the switch between the systems
and the storage system has to drop data. This happens because, while it hassingle connection
to the storage device, it has more traffic to send to the storage system than a single link can
carry. In this case, the switch drops network packets because the amount of data it can
transmit is limited by the speed of the link between it and the storage system.
Dropped Packets
iSCSI traffic is carried on the network by the Transmission Control Protocol (TCP). TCP is a
reliable transmission protocol that ensures that dropped packets are retried and eventually
reach their destination. TCP is designed to recover from dropped packets and retransmits
them quickly and seamlessly. However, when the switch discards packets with any
regularity, network throughput suffers significantly. The network becomes congested with
requests to resend data and with the resent packets, and less data is actually transferred than
in a network without congestion.
Most Ethernet switches can buffer, or store, data and give every device attempting to send
data an equal chance to get to the destination. This ability to buffer some transmissions,
combined with many systems limiting the number of outstanding commands, allows small
bursts from several systems to be sent to a storage system in turn.
If the transactions are large and multiple servers are trying to send data through a single
switch port, a switch's ability to buffer one request while another is transmitted can be
exceeded. In this case, the switch drops the data it cannot send, and the storage system must
request retransmission of the dropped packet. For example, if an Ethernet switch can buffer
32KB on an input port, but the server connected to it thinks it can send 256KB to the storage
device, some of the data is dropped.
Most managed switches provide information on dropped packets, similar to the following:
*: interface is up
IHQ: pkts in input hold queue
OHQ: pkts in output hold queue
RXBS: rx rate (bits/sec)
TXBS: tx rate (bits/sec)
TRTL: throttle count
Using VLANs or VPNs does not provide a suitable solution to the problem of link
oversubscription in shared configurations. VLANs and other virtual partitioning of a network
provide a way of logically designing a network, but do not change the physical capabilities of
links and trunks between switches. When storage traffic and other network traffic end up
sharing physical connections, as they would with a VPN, the possibility for oversubscription
and lost packets exists. The same is true of VLANs that share interswitch trunks.
Performance design for a SANs must take into account the physical limitations of the
network, not logical allocations.
All Paths A condition that occurs when a storage device becomes inaccessible to the host
Down
(APD)
and no paths to the device are available. ESXi treats this as a transient condition
because typically the problems with the device are temporary and the device is
expected to become available again.
Note
The sense codes must be received on all paths to the device for the device to be considered
permanently lost.
The following VMkernel log example of a SCSI sense code indicates that the device is in the
PDL state.
H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 or Logical Unit Not
Supported
For information about SCSI sense codes, see Troubleshooting Storage in vSphere
Troubleshooting.
In the case of iSCSI arrays with a single LUN per target, PDL is detected through iSCSI login
failure. An iSCSI storage array rejects your host's attempts to start an iSCSI session with a
reason Target Unavailable. As with the sense codes, this response must be received on all
paths for the device to be considered permanently lost.
After registering the PDL state of the device, the host stops attempts to reestablish
connectivity or to issue commands to the device to avoid becoming blocked or unresponsive.
The I/O from virtual machines is terminated.
Note
vSphere HA can detect PDL and restart failed virtual machines.
The vSphere Web Client displays the following information for the device:
It is possible for a device to return from PDL, however, data consistency is not guaranteed.
Note
The host cannot detect PDL conditions and continues to treat the device connectivity
problems as APD when a storage device permanently fails in a way that does not return
appropriate SCSI sense codes or a login rejection.
In contrast with the permanent device loss (PDL) state, the host treats the APD state as
transient and expects the device to be available again.
The host indefinitely continues to retry issued commands in an attempt to reestablish
connectivity with the device. If the host's commands fail the retries for a prolonged period of
time, the host and its virtual machines might be at risk of having performance problems and
potentially becoming unresponsive.
To avoid these problems, your host uses a default APD handling feature. When a device
enters the APD state, the system immediately turns on a timer and allows your host to
continue retrying nonvirtual machine commands for a limited time period.
By default, the APD timeout is set to 140 seconds, which is typically longer than most devices
need to recover from a connection loss. If the device becomes available within this time, the
host and its virtual machine continue to run without experiencing any problems.
If the device does not recover and the timeout ends, the host stops its attempts at retries and
terminates any nonvirtual machine I/O. Virtual machine I/O will continue retrying. The
vSphere Web Client displays the following information for the device with the expired APD
timeout:
Even though the device and datastores are unavailable, virtual machines remain responsive.
You can power off the virtual machines or migrate them to a different datastore or host.
If later one or more device paths becomes operational, subsequent I/O to the device is issued
normally and all special APD treatment ends.
5.14.3.1 Disable Storage APD Handling
The storage all paths down (APD) handling on your ESXi host is enabled by default. When it
is enabled, the host continues to retry nonvirtual machine I/O commands to a storage device
in the APD state for a limited time period. When the time period expires, the host stops its
retry attempts and terminates any nonvirtual machine I/O. You can disable the APD
handling feature on your host.
If you disable the APD handling, the host will indefinitely continue to retry issued commands
in an attempt to reconnect to the APD device. Continuing to retry is the same behavior as in
ESXi version 5.0. This behavior might cause virtual machines on the host to exceed their
internal I/O timeout and become unresponsive or fail. The host might become disconnected
from vCenter Server.
Procedure
If you disabled the APD handling, you can reenable it when a device enters the APD state.
The internal APD handling feature turns on immediately and the timer starts with the
current timeout value for each device in APD.
5.14.3.2 Change Timeout Limits for Storage APD
The timeout parameter controls how many seconds the ESXi host will retry nonvirtual
machine I/O commands to a storage device in an all paths down (APD) state. If needed, you
can change the default timeout value.
The timer starts immediately after the device enters the APD state. When the timeout
expires, the host marks the APD device as unreachable and fails any pending or new
nonvirtual machine I/O. Virtual machine I/O will continue to be retried.
The default timeout parameter on your host is 140 seconds. You can increase the value of the
timeout if, for example, storage devices connected to your ESXi host take longer than 140
seconds to recover from a connection loss.
Note
If you change the timeout value while an APD is in progress, it will not effect the timeout for
that APD.
Procedure
Procedure
1. Run the esxcli --server=server_name storage core device list d=device_ID command.
2. Check the connection status in the Status: field.
o on - Device is connected.
o dead - Device has entered the APD state. The APD timer starts.
o dead timeout - The APD timeout has expired.
o not connected - Device is in the PDL state.
5.14.5 PDL Conditions and High Availability
When a datastore enters a Permanent Device Loss (PDL) state, High Availability (HA) can
power off virtual machines located on the datastore and then restart them on an available
datastore. VMware offers advanced options to regulate the power off and restart operations
for virtual machines.
Advanced Parameters to Regulate PDL
Parameter
Description
disk.terminateVMOnPDLDefault
When set to true, this option enables default power off for
all virtual machines on the ESXi host.
scsi0:0.terminateVMOnPDL
Power off parameter that you can set for each individual
virtual machine.
This option overrides disk.terminateVMOnPDLDefault.
Use datastores that are created on SSD storage devices to allocate space for ESXi host
cache. For more information see the vSphere Resource Management documentation.
Make sure to use the latest firmware with SSD devices. Frequently check with your
storage vendors for any updates.
Carefully monitor how intensively you use the SSD device and calculate its estimated
lifetime. The lifetime expectancy depends on how actively you continue to use the
SSD device.
Typically, storage vendors provide reliable lifetime estimates for an SSD under ideal
conditions. For example, a vendor might guarantee a lifetime of 5 years under the condition
of 20GB writes per day. However, the more realistic life expectancy of the SSD will depend
on how many writes per day your host actually generates. Follow these steps to calculate the
lifetime of the SSD.
Procedure
Obtain the number of writes on the SSD by running the esxcli storage core
device stats get -d=device_ID command.
The Write Operations item in the output shows the number. You can average this
number over a period of time.
Estimate lifetime of your SSD by using the following formula:
vendor provided number of writes per day times vendor provided life span divided
by actual average number of writes per day
For example, if your vendor guarantees a lifetime of 5 years under the condition of
20GB writes per day, and the actual number of writes per day is 30GB, the life span
of your SSD will be approximately 3.3 years.
For information about block size limitations of a VMFS datastore, see the VMware
knowledge base article at http://kb.vmware.com/kb/1003565.
For VMFS3 datastores, the 2TB limit still applies, even when the storage device has a
capacity of more than 2TB. To be able to use the entire storage space, upgrade a
VMFS3 datastore to VMFS5. Conversion of the MBR format to GPT happens only
after you expand the datastore to a size larger than 2TB.
When you upgrade a VMFS3 datastore to VMFS5, the datastore uses the MBR
format. Conversion to GPT happens only after you expand the datastore to a size
larger than 2TB.
When you upgrade a VMFS3 datastore, remove from the storage device any partitions
that ESXi does not recognize, for example, partitions that use the EXT2 or EXT3
formats. Otherwise, the host cannot format the device with GPT and the upgrade
fails.
You cannot expand a VMFS3 datastore on devices that have the GPT partition
format.
Note
Always have only one VMFS datastore for each LUN.
You can store multiple virtual machines on the same VMFS datastore. Each virtual machine,
encapsulated in a set of files, occupies a separate single directory. For the operating system
inside the virtual machine, VMFS preserves the internal file system semantics, which ensures
correct application behavior and data integrity for applications running in virtual machines.
When you run multiple virtual machines, VMFS provides specific locking mechanisms for
virtual machine files, so that virtual machines can operate safely in a SAN environment
where multiple ESXi hosts share the same VMFS datastore.
In addition to virtual machines, the VMFS datastores can store other files, such as virtual
machine templates and ISO images.
When metadata changes are made in a shared storage enviroment, VMFS uses special
locking mechanisms to protect its data and prevent multiple hosts from concurrently writing
to the metadata.
5.15.6 VMFS Locking Mechanisms
In a shared storage environment, when multiple hosts access the same VMFS datastore,
specific locking mechanisms are used. These locking mechanism prevent multiple hosts from
concurrently writing to the metadata and ensure that no data corruption occurs.
VMFS supports SCSI reservations and atomic test and set (ATS) locking.
5.15.6.1 SCSI Reservations
VMFS uses SCSI reservations on storage devices that do not support hardware acceleration.
SCSI reservations lock an entire storage device while an operation that requires metadata
protection is performed. After the operation completes, VMFS releases the reservation and
other operations can continue. Because this lock is exclusive, excessive SCSI reservations by
a host can cause performance degradation on other hosts that are accessing the same VMFS.
For information about how to reduce SCSI reservations, see the vSphere Troubleshooting
documentation.
5.15.6.2 Atomic Test and Set (ATS)
For storage devices that support hardware acceleration, VMFS uses the ATS algorithm, also
called hardware assisted locking. In contrast with SCSI reservations, ATS supports discrete
locking per disk sector. For information about hardware acceleration, see Storage Hardware
Acceleration.
Mechanisms that VMFS uses to apply different types of locking depend on the VMFS
version.
Use of ATS Locking on Devices with Hardware Acceleration Support
Storage
Devices
New VMFS5
Upgraded VMFS5
VMFS3
ATS, but can revert to SCSI ATS, but can revert to SCSI
reservations
reservations
Multiple
extents
In certain cases, you might need to turn off the ATS-only setting for a new VMFS5 datastore.
For information, see Turn off ATS Locking.
5.15.6.3 Turn off ATS Locking
When you create a VMFS5 datastore on a device that supports atomic test and set (ATS)
locking, the datastore is set to the ATS-only mode. In certain circumstances, you might need
to turn off the ATS mode setting.
Turn off the ATS setting when, for example, your storage device is downgraded or firmware
updates fail and the device no longer supports hardware acceleration. The option that you
use to turn off the ATS setting is available only through the ESXi Shell. For more
information, see the Getting Started with vSphere Command-Line Interfaces.
Procedure
The device parameter is the path to the head extent device on which VMFS5 was
deployed. Use the following format:
/vmfs/devices/disks/disk_ID:P
Use Cisco's Hot Standby Router Protocol (HSRP) in IP Router. If you are using nonCisco router, be sure to use Virtual Router Redundancy Protocol (VRRP) instead.
Use Quality of Service (QoS) to prioritize NFS L3 traffic on networks with limited
bandwidths, or on networks that experience congestion. See your router
documentation for details.
Follow Routed NFS L3 best practices recommended by storage vendor. Contact your
storage vendor for details.
Disable Network I/O Resource Management (NetIORM).
If you are planning to use systems with top-of-rack switches or switch-dependent I/O
device partitioning, contact your system vendor for compatibility and support.
To upgrade a VMFS2 datastore, you use a two-step process that involves upgrading
VMFS2 to VMFS3 first. Because ESXi 5.0 and later hosts cannot access VMFS2
datastores, use a legacy host, ESX/ESXi 4.x or earlier, to access the VMFS2 datastore
and perform the VMFS2 to VMFS3 upgrade.
After you upgrade your VMFS2 datastore to VMFS3, the datastore becomes available
on the ESXi 5.x host, where you complete the process of upgrading to VMFS5.
You can perform a VMFS3 to VMFS5 upgrade while the datastore is in use with
virtual machines powered on.
While performing an upgrade, your host preserves all files on the datastore.
The datastore upgrade is a one-way process. After upgrading your datastore, you
cannot revert it back to its previous VMFS format.
Formatted
VMFS5
1, 2, 4, and 8MB
1MB
Subblock size
64KB
8KB
Partition format
Add a new extent. An extent is a partition on a storage device. You can add up to 32
extents of the same storage type to an existing VMFS datastore. The spanned VMFS
datastore can use any or all of its extents at any time. It does not need to fill up a
particular extent before using the next one.
Grow an extent in an existing VMFS datastore, so that it fills the available adjacent
capacity. Only extents with free space immediately after them are expandable.
Note
If a shared datastore has powered on virtual machines and becomes 100% full, you can
increase the datastore's capacity only from the host with which the powered on virtual
machines are registered.
Procedure
Value
scsi#.returnNoConnectDuringAPD
scsi#.returnBusyOnNoConnectStatus
e. Click OK.
To mange the hosts diagnostic partition, use the vCLI commands. See vSphere CommandLine Interface Concepts and Examples.
5.18.1 Create a Diagnostic Partition in the vSphere Client
You can create a diagnostic partition for your host.
Procedure
Log in to the vSphere Client and select the host from the Inventory panel.
Click the Configuration tab and click Storage in the Hardware panel.
Click Datastores and click Add Storage.
Select Diagnostic and click Next.
If you do not see Diagnostic as an option, the host already has a diagnostic partition.
True
False
device. The RDM contains metadata for managing and redirecting disk access to the physical
device.
The file gives you some of the advantages of direct access to a physical device while keeping
some advantages of a virtual disk in VMFS. As a result, it merges VMFS manageability with
raw device access.
RDMs can be described in terms such as mapping a raw device into a datastore, mapping a
system LUN, or mapping a disk file to a physical disk volume. All these terms refer to RDMs.
Raw Device Mapping
Although VMware recommends that you use VMFS datastores for most virtual disk storage,
on certain occasions, you might need to use raw LUNs or logical disks located in a SAN.
For example, you need to use raw LUNs with RDMs in the following situations:
When SAN snapshot or other layered applications run in the virtual machine. The
RDM better enables scalable backup offloading systems by using features inherent to
the SAN.
In any MSCS clustering scenario that spans physical hosts virtual-to-virtual
clusters as well as physical-to-virtual clusters. In this case, cluster data and quorum
disks should be configured as RDMs rather than as virtual disks on a shared VMFS.
Think of an RDM as a symbolic link from a VMFS volume to a raw LUN. The mapping makes
LUNs appear as files in a VMFS volume. The RDM, not the raw LUN, is referenced in the
virtual machine configuration. The RDM contains a reference to the raw LUN.
Using RDMs, you can:
Virtual compatibility mode allows an RDM to act exactly like a virtual disk file,
including the use of snapshots.
The RDM is not available for direct-attached block devices or certain RAID devices.
The RDM uses a SCSI serial number to identify the mapped device. Because block
devices and some direct-attach RAID devices do not export serial numbers, they
cannot be used with RDMs.
If you are using the RDM in physical compatibility mode, you cannot use a snapshot
with the disk. Physical compatibility mode allows the virtual machine to manage its
own, storage-based, snapshot or mirroring operations.
Virtual machine snapshots are available for RDMs with virtual compatibility mode.
You cannot map to a disk partition. RDMs require the mapped device to be a whole
LUN.
If you use vMotion to migrate virtual machines with RDMs, make sure to maintain
consistent LUN IDs for RDMs across all participating ESXi hosts.
You cannot relocate larger than 2TB RDMs to datastores other than VMFS5.
You cannot convert larger than 2TB RDMs to virtual disks, or perform other
operations that involve RDM to virtual disk conversion. Such operations include
cloning.
Virtual Disk
File
Virtual Mode
RDM
No
No
Yes
Yes
Yes
Snapshots
Yes
Yes
No
Distributed Locking
Yes
Yes
Yes
Clustering
Cluster-in-a-box
Physical-to-virtual clustering
Cluster-in-a-box
cluster-acrossonly
cluster-across-boxes
boxes
SCSI Target-Based
Software
No
No
Yes
VMware recommends that you use virtual disk files for the cluster-in-a-box type of
clustering. If you plan to reconfigure your cluster-in-a-box clusters as cluster-across-boxes
clusters, use virtual mode RDMs for the cluster-in-a-box clusters.
Option
Description
Persistent
NonpersistentChanges to the disk are discarded when you power off or revert to the
snapshot.
The primary file system that the VMkernel uses is the VMware Virtual Machine File System
(VMFS). VMFS is a cluster file system designed and optimized to support large files such as
virtual disks and swap files. The VMkernel also supports the storage of virtual disks on NFS
file systems.
The storage I/O path provides virtual machines with access to storage devices through device
emulation. This device emulation allows a virtual machine to access files on a VMFS or NFS
file system as if they were SCSI devices. The VMkernel provides storage virtualization
functions such as the scheduling of I/O requests from multiple virtual machines and
multipathing.
In addition, VMkernel offers several Storage APIs that enable storage partners to integrate
and optimize their products for vSphere.
The following graphic illustrates the basics of the VMkernel core, with special attention to
the storage stack. Storagerelated modules reside between the logical device I/O scheduler
and the adapter I/O scheduler layers.
VMkernel and Storage
Storage APIs
Storage APIs - Multipathing, also known as the Pluggable Storage Architecture (PSA).
PSA is a collection of VMkernel APIs that allows storage partners to enable and
certify their arrays asynchronous to ESXi release schedules, as well as deliver
performanceenhancing, multipathing and loadbalancing behaviors that are
optimized for each array. For more information, see Managing Multiple Paths.
Storage APIs - Array Integration, formerly known as VAAI, include the following
APIs:
o Hardware Acceleration APIs. Allows arrays to integrate with vSphere to
transparently offload certain storage operations to the array. This integration
significantly reduces CPU overhead on the host. See Storage Hardware
Acceleration.
o Array Thin Provisioning APIs. Help to monitor space use on thin-provisioned
storage arrays to prevent out-of-space conditions, and to perform space
reclamation. See Array Thin Provisioning and VMFS Datastores.
Storage APIs - Storage Awareness. These vCenter Server-based APIs enable storage
arrays to inform the vCenter Server about their configurations, capabilities, and
storage health and events. See Using Storage Vendor Providers.
ESXi does not support multipathing when you combine an independent hardware
adapter with software iSCSI or dependent iSCSI adapters in the same host.
Multipathing between software and dependent adapters within the same host is
supported.
On different hosts, you can mix both dependent and independent adapters.
The following illustration shows multipathing setups possible with different types of iSCSI
initiators.
Host-Based Path Failover
When using one of these storage systems, your host does not see multiple ports on the
storage and cannot choose the storage port it connects to. These systems have a single virtual
port address that your host uses to initially communicate. During this initial communication,
the storage system can redirect the host to communicate with another port on the storage
system. The iSCSI initiators in the host obey this reconnection request and connect with a
different port on the system. The storage system uses this technique to spread the load
across available ports.
If the ESXi host loses connection to one of these ports, it automatically attempts to reconnect
with the virtual port of the storage system, and should be redirected to an active, usable port.
This reconnection and redirection happens quickly and generally does not disrupt running
virtual machines. These storage systems can also request that iSCSI initiators reconnect to
the system, to change which storage port they are connected to. This allows the most
effective use of the multiple ports.
The Port Redirection illustration shows an example of port redirection. The host attempts to
connect to the 10.0.0.1 virtual port. The storage system redirects this request to 10.0.0.2. The
host connects with 10.0.0.2 and uses this port for I/O communication.
Note
The storage system does not always redirect connections. The port at 10.0.0.1 could be used
for traffic, also.
Port Redirection
If the port on the storage system that is acting as the virtual port becomes unavailable, the
storage system reassigns the address of the virtual port to another port on the system. Port
Reassignment shows an example of this type of port reassignment. In this case, the virtual
port 10.0.0.1 becomes unavailable and the storage system reassigns the virtual port IP
address to a different port. The second port responds to both addresses.
Port Reassignment
With this form of array-based failover, you can have multiple paths to the storage only if you
use multiple ports on the ESXi host. These paths are active-active. For additional
information, see iSCSI Session Management.
NMP
PSP
Path Selection Plug-In, also called Path Selection Policy. Handles path selection
for a given device.
SATP
Storage Array Type Plug-In, also called Storage Array Type Policy. Handles path
failover for a given storage array.
The VMkernel multipathing plug-in that ESXi provides by default is the VMware Native
Multipathing Plug-In (NMP). The NMP is an extensible module that manages sub plug-ins.
There are two types of NMP sub plug-ins, Storage Array Type Plug-Ins (SATPs), and Path
Selection Plug-Ins (PSPs). SATPs and PSPs can be built-in and provided by VMware, or can
be provided by a third party.
If more multipathing functionality is required, a third party can also provide an MPP to run
in addition to, or as a replacement for, the default NMP.
When coordinating the VMware NMP and any installed third-party MPPs, the PSA performs
the following tasks:
As the Pluggable Storage Architecture illustration shows, multiple third-party MPPs can run
in parallel with the VMware NMP. When installed, the third-party MPPs replace the
behavior of the NMP and take complete control of the path failover and the load-balancing
operations for specified storage devices.
After the NMP determines which SATP to use for a specific storage device and associates the
SATP with the physical paths for that storage device, the SATP implements the tasks that
include the following:
The host selects the path that it used most recently. When the path
becomes unavailable, the host selects an alternative path. The host does
not revert back to the original path when that path becomes available
again. There is no preferred path setting with the MRU policy. MRU is
the default policy for most active-passive storage devices.
The VMW_PSP_MRU ranking capability allows you to assign ranks to
individual paths. To set ranks to individual paths, use the esxcli storage
nmp psp generic pathconfig set command. For details, see the VMware
knowledge base article at http://kb.vmware.com/kb/2003468.
The policy is displayed in the client as the Most Recently Used
(VMware) path selection policy.
VMW_PSP_FIXED The host uses the designated preferred path, if it has been configured.
Otherwise, it selects the first working path discovered at system boot
time. If you want the host to use a particular preferred path, specify it
manually. Fixed is the default policy for most active-active storage
devices.
Note
If the host uses a default preferred path and the path's status turns to
Dead, a new path is selected as preferred. However, if you explicitly
designate the preferred path, it will remain preferred even when it
becomes inaccessible.
VMW_PSP_RR
The host uses an automatic path selection algorithm rotating through all
active paths when connecting to active-passive arrays, or through all
available paths when connecting to active-active arrays. RR is the
default for a number of arrays and can be used with both active-active
and active-passive arrays to implement load balancing across paths for
different LUNs.
Displayed in the client as the Round Robin (VMware) path selection
policy.
Use the vSphere Client or the vSphere Web Client to view which SATP and PSP the host is
using for a specific storage device and the status of all available paths for this storage device.
If needed, you can change the default VMware PSP using the client. To change the default
SATP, you need to modify claim rules using the vSphere CLI.
You can find some information about modifying claim rules in Managing Storage Paths and
Multipathing Plug-Ins.
For more information about the commands available to manage PSA, see Getting Started
with vSphere Command-Line Interfaces.
For a complete list of storage arrays and corresponding SATPs and PSPs, see the SAN Array
Model Reference section of the vSphere Compatibility Guide.
5.27.1 Viewing the Paths Information
You can review the storage array type policy (SATP) and path selection policy (PSP) that the
ESXi host uses for a specific storage device and the status of all available paths for this
storage device. You can access the path information from both the Datastores and Devices
views. For datastores, you review the paths that connect to the device the datastore is
deployed on.
The path information includes the SATP assigned to manage the device, the PSP, a list of
paths, and the status of each path. The following path status information can appear:
Active
Paths available for issuing I/O to a LUN. A single or multiple working paths
currently used for transferring data are marked as Active (I/O).
Standby If active paths fail, the path can quickly become operational and can be used for
I/O.
Disabled The path is disabled and no data can be transferred.
Dead
If you are using the Fixed path policy, you can see which path is the preferred path. The
preferred path is marked with an asterisk (*) in the Preferred column.
For each path you can also display the path's name. The name includes parameters that
describe the path: adapter ID, target ID, and device ID. Usually, the path's name has the
format similar to the following:
fc.adapterID-fc.targetID-naa.deviceID
Note
When you use the host profiles editor to edit paths, you must specify all three parameters
that describe a path, adapter ID, target ID, and device ID
You can display all multipathing plug-ins available on your host. You can list any third-party
MPPs, as well as your host's NMP and SATPs and review the paths they claim. You can also
define new paths and specify which multipathing plug-in should claim the paths.
For more information about commands available to manage PSA, see the Getting Started
with vSphere Command-Line Interfaces.
5.29Multipathing Considerations
Specific considerations apply when you manage storage multipathing plug-ins and claim
rules.
The following considerations help you with multipathing:
If no SATP is assigned to the device by the claim rules, the default SATP for iSCSI or
FC devices is VMW_SATP_DEFAULT_AA. The default PSP is VMW_PSP_FIXED.
When the system searches the SATP rules to locate a SATP for a given device, it
searches the driver rules first. If there is no match, the vendor/model rules are
searched, and finally the transport rules are searched. If no match occurs, NMP
selects a default SATP for the device.
If VMW_SATP_ALUA is assigned to a specific storage device, but the device is not
ALUA-aware, no claim rule match occurs for this device. The device is claimed by the
default SATP based on the device's transport type.
The default PSP for all devices claimed by VMW_SATP_ALUA is VMW_PSP_MRU.
The VMW_PSP_MRU selects an active/optimized path as reported by the
VMW_SATP_ALUA, or an active/unoptimized path if there is no active/optimized
path. This path is used until a better path is available (MRU). For example, if the
VMW_PSP_MRU is currently using an active/unoptimized path and an
active/optimized path becomes available, the VMW_PSP_MRU will switch the
current path to the active/optimized one.
While VMW_PSP_MRU is typically selected for ALUA arrays by default, certain
ALUA storage arrays need to use VMW_PSP_FIXED. To check whether your storage
array requires VMW_PSP_FIXED, see the VMware Compatibility Guide or contact
your storage vendor. When using VMW_PSP_FIXED with ALUA arrays, unless you
explicitly specify a preferred path, the ESXi host selects the most optimal working
path and designates it as the default preferred path. If the host selected path becomes
unavailable, the host selects an alternative available path. However, if you explicitly
designate the preferred path, it will remain preferred no matter what its status is.
By default, the PSA claim rule 101 masks Dell array pseudo devices. Do not delete this
rule, unless you want to unmask these devices.
Vendor/model strings
Transportation, such as SATA, IDE, Fibre Channel, and so on
Adapter, target, or LUN location
Device driver, for example, Mega-RAID
In the procedure, --server=server_name specifies the target server. The specified target
server prompts you for a user name and password. Other connection options, such as a
configuration file or session file, are supported. For a list of connection options, see Getting
Started with vSphere Command-Line Interfaces.
Prerequisites
Install vCLI or deploy the vSphere Management Assistant (vMA) virtual machine. See
Getting Started with vSphere Command-Line Interfaces. For troubleshooting , run esxcli
commands in the ESXi Shell.
Procedure
1. Run the esxcli --server=server_name storage core claimrule list -claimrule-class=MP command to list the multipathing claim rules.
5.29.1.1 Example: Sample Output of the esxcli storage core claimrule list Command
Rule Class Rule
Class
MP
0
runtime
MP
1
runtime
MP
2
runtime
MP
3
runtime
MP
4
runtime
MP
101
runtime
model=Universal Xport
MP
101
file
model=Universal Xport
MP
200
runtime
MP
200
file
MP
201
runtime
target=* lun=*
MP
201
file
target=* lun=*
MP
202
runtime
MP
202
file
MP
65535 runtime
Type
transport
transport
transport
transport
transport
vendor
Plugin
NMP
NMP
NMP
NMP
NMP
MASK_PATH
Matches
transport=usb
transport=sata
transport=ide
transport=block
transport=unknown
vendor=DELL
vendor
MASK_PATH
vendor=DELL
vendor
vendor
location
MPP_1
MPP_1
MPP_2
vendor=NewVend model=*
vendor=NewVend model=*
adapter=vmhba41 channel=*
location
MPP_2
adapter=vmhba41 channel=*
driver
driver
vendor
MPP_3
MPP_3
NMP
driver=megaraid
driver=megaraid
vendor=* model=*
The NMP claims all paths connected to storage devices that use the USB, SATA, IDE,
and Block SCSI transportation.
You can use the MASK_PATH module to hide unused devices from your host. By
default, the PSA claim rule 101 masks Dell array pseudo devices with a vendor string
of DELL and a model string of Universal Xport.
The MPP_1 module claims all paths connected to any model of the NewVend storage
array.
The MPP_3 module claims the paths to storage devices controlled by the Mega-RAID
device driver.
Any paths not described in the previous rules are claimed by NMP.
The Rule Class column in the output describes the category of a claim rule. It can
Full copy, also called clone blocks or copy offload. Enables the storage arrays to make
full copies of data within the array without having the host read and write the data.
This operation reduces the time and network load when cloning virtual machines,
provisioning from a template, or migrating with vMotion.
Block zeroing, also called write same. Enables storage arrays to zero out a large
number of blocks to provide newly allocated storage, free of previously written data.
This operation reduces the time and network load when creating virtual machines
and formatting virtual disks.
Hardware assisted locking, also called atomic test and set (ATS). Supports discrete
virtual machine locking without use of SCSI reservations. This operation allows disk
locking per sector, instead of the entire LUN as with SCSI reservations.
Check with your vendor for the hardware acceleration support. Certain storage arrays
require that you activate the support on the storage side.
On your host, the hardware acceleration is enabled by default. If your storage does not
support the hardware acceleration, you can disable it.
In addition to hardware acceleration support, ESXi includes support for array thin
provisioning. For information, see Array Thin Provisioning and VMFS Datastores.
ESXi host can communicate directly and does not require the VAAI plug-ins.
If the device does not support T10 SCSI or provides partial support, ESXi reverts to using the
VAAI plug-ins, installed on your host, or uses a combination of the T10 SCSI commands and
plug-ins. The VAAI plug-ins are vendor-specific and can be either VMware or partner
developed. To manage the VAAI capable device, your host attaches the VAAI filter and
vendor-specific VAAI plug-in to the device.
For information about whether your storage requires VAAI plug-ins or supports hardware
acceleration through T10 SCSI commands, see the vSphere Compatibility Guide or check
with your storage vendor.
You can use several esxcli commands to query storage devices for the hardware acceleration
support information. For the devices that require the VAAI plug-ins, the claim rule
commands are also available. For information about esxcli commands, see Getting Started
with vSphere Command-Line Interfaces.
5.29.2.1 Display Hardware Acceleration Plug-Ins and Filter
To communicate with the devices that do not support the T10 SCSI standard, your host uses
a combination of a single VAAI filter and a vendor-specific VAAI plug-in. Use the esxcli
command to view the hardware acceleration filter and plug-ins currently loaded into your
system.
In the procedure, --server=server_name specifies the target server. The specified target
server prompts you for a user name and password. Other connection options, such as a
configuration file or session file, are supported. For a list of connection options, see Getting
Started with vSphere Command-Line Interfaces.
Full file clone. This operation is similar to the VMFS block cloning except that NAS
devices clone entire files instead of file segments.
Reserve space. Enables storage arrays to allocate space for a virtual disk file in thick
format.
Typically, when you create a virtual disk on an NFS datastore, the NAS server determines the
allocation policy. The default allocation policy on most NAS servers is thin and does not
guarantee backing storage to the file. However, the reserve space operation can instruct the
NAS device to use vendor-specific mechanisms to reserve space for a virtual disk. As a result,
you can create thick virtual disks on the NFS datastore.
Lazy file clone. Allows VMware View to offload creation of linked clones to a NAS
array.
Extended file statistics. Enables storage arrays to accurately report space utilization.
With NAS storage devices, the hardware acceleration integration is implemented through
vendor-specific NAS plug-ins. These plug-ins are typically created by vendors and are
distributed as VIB packages through a web page. No claim rules are required for the NAS
plug-ins to function.
There are several tools available for installing and upgrading VIB packages. They include the
esxcli commands and vSphere Update Manager. For more information, see the vSphere
Upgrade and Installing and Administering VMware vSphere Update Manager
documentation.
The source and destination VMFS datastores have different block sizes.
The source file type is RDM and the destination file type is non-RDM (regular file).
The source VMDK type is eagerzeroedthick and the destination VMDK type is thin.
The source or destination VMDK is in sparse or hosted format.
Hardware cloning between arrays, even within the same VMFS datastore, does not work.
The configuration parameters are set in the option ROM of your adapter. During an ESXi
installation or a subsequent boot, these parameters are exported in to system memory in either
FBFT format or FBPT format. The VMkernel can read the configuration settings and use
them to access the boot LUN.
This chapter includes the following topics:
Contain FCoE boot firmware which can export boot information in FBFT
format or FBPT format.
5.33.2 Considerations
You cannot change software FCoE boot configuration from within ESXi.
Coredump is not supported on any software FCoE LUNs, including the boot LUN.
Multipathing is not supported at pre-boot.
Boot LUN cannot be shared with other hosts even on shared storage.
Make sure that the host has access to the entire boot LUN. The boot LUN cannot be
shared with other hosts even on shared storage.
If you use Intel 10 Gigabit Ethernet Controller (Niantec) with a Cisco switch,
configure the switch port in the following way:
o Enable the Spanning Tree Protocol (STP).
o Turn off switchport trunk native vlan for the VLAN used for FCoE.
High
Normal
Low
For example, an SMP virtual machine with two virtual CPUs and 1GB RAM with CPU and
memory shares set to Normal has 2x1000=2000 shares of CPU and 10x1024=10240 shares
of memory.
Note
Virtual machines with more than one virtual CPU are called SMP (symmetric
multiprocessing) virtual machines. ESXi supports up to 64 virtual CPUs per virtual machine.
The relative priority represented by each share changes when a new virtual machine is
powered on. This affects all virtual machines in the same resource pool. All of the irtual
machines have the same number of virtual CPUs. Consider the following examples.
Two CPU-bound virtual machines run on a host with 8GHz of aggregate CPU
capacity. Their CPU shares are set to Normal and get 4GHz each.
A third CPU-bound virtual machine is powered on. Its CPU shares value is set to
High, which means it should have twice as many shares as the machines set to
Normal. The new virtual machine receives 4GHz and the two other machines get only
2GHz each. The same result occurs if the user specifies a custom share value of 2000
for the third virtual machine.
Benefits Assigning a limit is useful if you start with a small number of virtual
machines and want to manage user expectations. Performance deteriorates as you
add more virtual machines. You can simulate having fewer resources available by
specifying a limit.
Drawbacks You might waste idle resources if you specify a limit. The system does
not allow virtual machines to use more resources than the limit, even when the
system is underutilized and idle resources are available. Specify the limit only if you
have good reasons for doing so.
If you expect frequent changes to the total available resources, use Shares to allocate
resources fairly across virtual machines. If you use Shares, and you upgrade the host,
for example, each virtual machine stays at the same priority (keeps the same number
of shares) even though each share represents a larger amount of memory, CPU, or
storage I/O resources.
Use Reservation to specify the minimum acceptable amount of CPU or memory, not
the amount you want to have available. The host assigns additional resources as
available based on the number of shares, estimated demand, and the limit for your
virtual machine. The amount of concrete resources represented by a reservation does
not change when you change the environment, such as by adding or removing virtual
machines.
When specifying the reservations for virtual machines, do not commit all resources
(plan to leave at least 10% unreserved). As you move closer to fully reserving all
capacity in the system, it becomes increasingly difficult to make changes to
reservations and to the resource pool hierarchy without violating admission control.
In a DRS-enabled cluster, reservations that fully commit the capacity of the cluster or
of individual hosts in the cluster can prevent DRS from migrating virtual machines
between hosts.
Option
Description
Shares
CPU shares for this resource pool with respect to the parents total. Sibling
resource pools share resources according to their relative share values bounded
by the reservation and limit. Select Low, Normal, or High, which specify share
values respectively in a 1:2:4 ratio. Select Custom to give each virtual machine a
specific number of shares, which expresses a proportional weight.
Upper limit for this resource pools CPU allocation. Select Unlimited to specify
no upper limit.
Description
Shares
Memory shares for this resource pool with respect to the parents total. Sibling
resource pools share resources according to their relative share values bounded
by the reservation and limit. Select Low, Normal, or High, which specify share
values respectively in a 1:2:4 ratio. Select Custom to give each virtual machine a
Limit
In the following example, assume that VM-QA is memory intensive and accordingly you
want to change the resource allocation settings for the two virtual machines to:
Specify that, when system memory is overcommitted, VM-QA can use twice as much
memory and CPU as the Marketing virtual machine. Set the memory shares and CPU
shares for VM-QA to High and for VM-Marketing set them to Normal.
Ensure that the Marketing virtual machine has a certain amount of guaranteed CPU
resources. You can do so using a reservation setting.
Procedure
If you select the clusters Resource Allocation tab and click CPU, you should see that shares
for VM-QA are twice that of the other virtual machine. Also, because the virtual machines
have not been powered on, the Reservation Used fields have not changed.
6.1.6 Admission Control
When you power on a virtual machine, the system checks the amount of CPU and memory
resources that have not yet been reserved. Based on the available unreserved resources, the
system determines whether it can guarantee the reservation for which the virtual machine is
configured (if any). This process is called admission control.
If enough unreserved CPU and memory are available, or if there is no reservation, the virtual
machine is powered on. Otherwise, an Insufficient Resources warning appears.
Note
In addition to the user-specified memory reservation, for each virtual machine there is also
an amount of overhead memory. This extra memory commitment is included in the
admission control calculation.
When the vSphere DPM feature is enabled, hosts might be placed in standby mode (that is,
powered off) to reduce power consumption. The unreserved resources provided by these
hosts are considered available for admission control. If a virtual machine cannot be powered
on without these resources, a recommendation to power on sufficient standby hosts is made.
6.1.7 CPU Virtualization Basics
CPU virtualization emphasizes performance and runs directly on the processor whenever
possible. The underlying physical resources are used whenever possible and the
virtualization layer runs instructions only as needed to make virtual machines operate as if
they were running directly on a physical machine.
CPU virtualization is not the same thing as emulation. ESXi does not use emulation to run
virtual CPUs. With emulation, all operations are run in software by an emulator. A software
emulator allows programs to run on a computer system other than the one for which they
were originally written. The emulator does this by emulating, or reproducing, the original
computers behavior by accepting the same data or inputs and achieving the same results.
Emulation provides portability and runs software designed for one platform across several
platforms.
When CPU resources are overcommitted, the ESXi host time-slices the physical processors
across all virtual machines so each virtual machine runs as if it has its specified number of
virtual processors. When an ESXi host runs multiple virtual machines, it allocates to each
virtual machine a share of the physical resources. With the default resource allocation
settings, all virtual machines associated with the same host receive an equal share of CPU per
virtual CPU. This means that a single-processor virtual machines is assigned only half of the
resources of a dual-processor virtual machine.
This chapter includes the following topics:
For applications that are not CPU-bound, CPU virtualization likely translates into an
increase in CPU use. If spare CPU capacity is available to absorb the overhead, it can still
deliver comparable performance in terms of overall throughput.
Note
Deploy single-threaded applications on uniprocessor virtual machines, instead of on SMP
virtual machines that have multiple CPUs, for the best performance and resource use.
Single-threaded applications can take advantage only of a single CPU. Deploying such
applications in dual-processor virtual machines does not speed up the application. Instead, it
causes the second virtual CPU to use physical resources that other virtual machines could
otherwise use.
Use the attributes and special features available through the vSphere Client. The
vSphere Client graphical user interface (GUI) allows you to connect to the ESXi host
or a vCenter Server system.
Use advanced settings under certain circumstances.
Use the vSphere SDK for scripted CPU allocation.
Use hyperthreading.
Each logical processor of each processor core can be used independently by the ESXi CPU
scheduler to execute virtual machines, providing capabilities similar to SMP systems. For
example, a two-way virtual machine can have its virtual processors running on logical
processors that belong to the same core, or on logical processors on different physical cores.
The ESXi CPU scheduler can detect the processor topology and the relationships between
processor cores and the logical processors on them. It uses this information to schedule
virtual machines and optimize performance.
The ESXi CPU scheduler can interpret processor topology, including the relationship
between sockets, cores, and logical processors. The scheduler uses topology information to
optimize the placement of virtual CPUs onto different sockets to maximize overall cache
utilization, and to improve cache affinity by minimizing virtual CPU migrations.
In undercommitted systems, the ESXi CPU scheduler spreads load across all sockets by
default. This improves performance by maximizing the aggregate amount of cache available
to the running virtual CPUs. As a result, the virtual CPUs of a single SMP virtual machine are
spread across multiple sockets (unless each socket is also a NUMA node, in which case the
NUMA scheduler restricts all the virtual CPUs of the virtual machine to reside on the same
socket.)
In some cases, such as when an SMP virtual machine exhibits significant data sharing
between its virtual CPUs, this default behavior might be sub-optimal. For such workloads, it
can be beneficial to schedule all of the virtual CPUs on the same socket, with a shared lastlevel cache, even when the ESXi host is undercommitted. In such scenarios, you can override
the default behavior of spreading virtual CPUs across packages by including the following
configuration option in the virtual machine's .vmx configuration file:
sched.cpu.vsmpConsolidate="TRUE".
6.1.13 Hyperthreading
Hyperthreading technology allows a single physical processor core to behave like two logical
processors. The processor can run two independent applications at the same time. To avoid
confusion between logical and physical processors, Intel refers to a physical processor as a
socket, and the discussion in this chapter uses that terminology as well.
Intel Corporation developed hyperthreading technology to enhance the performance of its
Pentium IV and Xeon processor lines. Hyperthreading technology allows a single processor
core to execute two independent threads simultaneously.
While hyperthreading does not double the performance of a system, it can increase
performance by better utilizing idle resources leading to greater throughput for certain
important workload types. An application running on one logical processor of a busy core
can expect slightly more than half of the throughput that it obtains while running alone on a
non-hyperthreaded processor. Hyperthreading performance improvements are highly
application-dependent, and some applications might see performance degradation with
hyperthreading because many processor resources (such as the cache) are shared between
logical processors.
Note
On processors with Intel Hyper-Threading technology, each core can have two logical
processors which share most of the core's resources, such as memory caches and functional
units. Such logical processors are usually called threads.
Many processors do not support hyperthreading and as a result have only one thread per
core. For such processors, the number of cores also matches the number of logical
processors. The following processors support hyperthreading and have two threads per core.
The default for all virtual machines on a hyperthreaded system. The virtual CPUs of
a virtual machine with this setting can freely share cores with other virtual CPUs
from this or any other virtual machine at any time.
None
Virtual CPUs of a virtual machine should not share cores with each other or with
virtual CPUs from other virtual machines. That is, each virtual CPU from this virtual
machine should always get a whole core to itself, with the other logical CPU on that
core being placed into the halted state.
This option is similar to none. Virtual CPUs from this virtual machine cannot share
cores with virtual CPUs from other virtual machines. They can share cores with the
Internalother virtual CPUs from the same virtual machine.
You can select this option only for SMP virtual machines. If applied to a
uniprocessor virtual machine, the system changes this option to none.
These options have no effect on fairness or CPU time allocation. Regardless of a virtual
machines hyperthreading settings, it still receives CPU time proportional to its CPU shares,
and constrained by its CPU reservation and CPU limit values.
For typical workloads, custom hyperthreading settings should not be necessary. The options
can help in case of unusual workloads that interact badly with hyperthreading. For example,
an application with cache thrashing problems might slow down an application sharing its
physical core. You can place the virtual machine running the application in the none or
internal hyperthreading status to isolate it from other virtual machines.
If a virtual CPU has hyperthreading constraints that do not allow it to share a core with
another virtual CPU, the system might deschedule it when other virtual CPUs are entitled to
consume processor time. Without the hyperthreading constraints, you can schedule both
virtual CPUs on the same core.
The problem becomes worse on systems with a limited number of cores (per virtual
machine). In such cases, there might be no core to which the virtual machine that is
descheduled can be migrated. As a result, virtual machines with hyperthreading set to none or
internal can experience performance degradation, especially on systems with a limited
number of cores.
6.1.16 Quarantining
In certain rare circumstances, ESXi might detect that an application is interacting badly with
the Pentium IV hyperthreading technology. (This does not apply to systems based on the
Intel Xeon 5500 processor microarchitecture.) In such cases, quarantining, which is
transparent to the user, might be necessary.
For example, certain types of self-modifying code can disrupt the normal behavior of the
Pentium IV trace cache and can lead to substantial slowdowns (up to 90 percent) for an
application sharing a core with the problematic code. In those cases, the ESXi host
quarantines the virtual CPU running this code and places its virtual machine in the none or
internal mode, as appropriate.
Before you use CPU affinity, you might need to consider certain issues.
Potential issues with CPU affinity include:
For multiprocessor systems, ESXi systems perform automatic load balancing. Avoid
manual specification of virtual machine affinity to improve the schedulers ability to
balance load across processors.
Affinity can interfere with the ESXi hosts ability to meet the reservation and shares
specified for a virtual machine.
Because CPU admission control does not consider affinity, a virtual machine with
manual affinity settings might not always receive its full reservation.
Virtual machines that do not have manual affinity settings are not adversely affected
by virtual machines with manual affinity settings.
When you move a virtual machine from one host to another, affinity might no longer
apply because the new host might have a different number of processors.
The NUMA scheduler might not be able to manage a virtual machine that is already
assigned to certain processors using affinity.
Affinity can affect the host's ability to schedule virtual machines on multicore or
hyperthreaded processors to take full advantage of resources shared on such
processors.
Description
Not supported
The host does not support any power management features or power
management is not enabled in the BIOS.
The VMkernel detects certain power management features, but will not
High Performance use them unless the BIOS requests them for power capping or thermal
events.
The VMkernel uses the available power management features
Balanced (Default) conservatively to reduce host energy consumption with minimal
compromise to performance.
Low Power
Custom
When a CPU runs at lower frequency, it can also run at lower voltage, which saves power.
This type of power management is typically called Dynamic Voltage and Frequency Scaling
(DVFS). ESXi attempts to adjust CPU frequencies so that virtual machine performance is not
affected.
When a CPU is idle, ESXi can take advantage of deep halt states (known as C-states). The
deeper the C-state, the less power the CPU uses, but the longer it takes for the CPU to
resume running. When a CPU becomes idle, ESXi applies an algorithm to predict how long it
will be in an idle state and chooses an appropriate C-state to enter. In power management
policies that do not use deep C-states, ESXi uses only the shallowest halt state (C1) for idle
CPUs.
6.1.20 Select a CPU Power Management Policy
You set the CPU power management policy for a host using the vSphere Client.
Prerequisites
Verify that the BIOS settings on the host system allow the operating system to control power
management (for example, OS Controlled).
Note
Some systems have Processor Clocking Control (PCC) technology, which allows ESXi to
manage power on the host system even if the host BIOS settings do not specify OS Controlled
mode. With this technology, ESXi does not manage P-states directly. Instead, the host
cooperates with the BIOS to determine the processor clock rate. HP systems that support
this technology have a BIOS setting called Cooperative Power Management that is enabled
by default.
The configured size is a construct maintained by the virtualization layer for the virtual
machine. It is the amount of memory that is presented to the guest operating system, but it is
independent of the amount of physical RAM that is allocated to the virtual machine, which
depends on the resource settings (shares, reservation, limit) explained below.
For example, consider a virtual machine with a configured size of 1GB. When the guest
operating system boots, it detects that it is running on a dedicated machine with 1GB of
physical memory. The actual amount of physical host memory allocated to the virtual
machine depends on its memory resource settings and memory contention on the ESXi host.
In some cases, the virtual machine might be allocated the full 1GB. In other cases, it might
receive a smaller allocation. Regardless of the actual allocation, the guest operating system
continues to behave as though it is running on a dedicated machine with 1GB of physical
memory.
Shares
Specify the relative priority for a virtual machine if more than the reservation is
available.
Reservation Is a guaranteed lower bound on the amount of physical memory that the host
reserves for the virtual machine, even when memory is overcommitted. Set the
reservation to a level that ensures the virtual machine has sufficient memory to
run efficiently, without excessive paging.
After a virtual machine has accessed its full reservation, it is allowed to retain
that amount of memory and this memory is not reclaimed, even if the virtual
machine becomes idle. For example, some guest operating systems (for
example, Linux) might not access all of the configured memory immediately
after booting. Until the virtual machines accesses its full reservation, VMkernel
can allocate any unused portion of its reservation to other virtual machines.
However, after the guests workload increases and it consumes its full
reservation, it is allowed to keep this memory.
Limit Is an upper bound on the amount of physical memory that the host can allocate to the
virtual machine. The virtual machines memory allocation is also implicitly limited by
its configured size.
Overhead memory includes space reserved for the virtual machine frame buffer and
various virtualization data structures.
6.2.2 Memory Overcommitment
For each running virtual machine, the system reserves physical memory for the virtual
machines reservation (if any) and for its virtualization overhead.
Because of the memory management techniques the ESXi host uses, your virtual machines
can use more memory than the physical machine (the host) has available. For example, you
can have a host with 2GB memory and run four virtual machines with 1GB memory each. In
that case, the memory is overcommitted.
Overcommitment makes sense because, typically, some virtual machines are lightly loaded
while others are more heavily loaded, and relative activity levels vary over time.
To improve memory utilization, the ESXi host transfers memory from idle virtual machines
to virtual machines that need more memory. Use the Reservation or Shares parameter to
preferentially allocate memory to important virtual machines. This memory remains
available to other virtual machines if it is not in use.
In addition, memory compression is enabled by default on ESXi hosts to improve virtual
machine performance when memory is overcommitted as described in Memory
Compression.
6.2.3 Memory Sharing
Many workloads present opportunities for sharing memory across virtual machines.
For example, several virtual machines might be running instances of the same guest
operating system, have the same applications or components loaded, or contain common
data. ESXi systems use a proprietary page-sharing technique to securely eliminate redundant
copies of memory pages.
With memory sharing, a workload consisting of multiple virtual machines often consumes
less memory than it would when running on physical machines. As a result, the system can
efficiently support higher levels of overcommitment.
The amount of memory saved by memory sharing depends on workload characteristics. A
workload of many nearly identical virtual machines might free up more than thirty percent of
memory, while a more diverse workload might result in savings of less than five percent of
memory.
6.2.4 Software-Based Memory Virtualization
ESXi virtualizes guest physical memory by adding an extra level of address translation.
The VMM for each virtual machine maintains a mapping from the guest operating
system's physical memory pages to the physical memory pages on the underlying
machine. (VMware refers to the underlying host physical pages as machine pages
and the guest operating systems physical pages as physical pages.)
Each virtual machine sees a contiguous, zero-based, addressable physical memory space. The
underlying machine memory on the server used by each virtual machine is not necessarily
contiguous.
The VMM intercepts virtual machine instructions that manipulate guest operating
system memory management structures so that the actual memory management unit
(MMU) on the processor is not updated directly by the virtual machine.
The ESXi host maintains the virtual-to-machine page mappings in a shadow page
table that is kept up to date with the physical-to-machine mappings (maintained by
the VMM).
The shadow page tables are used directly by the processor's paging hardware.
This approach to address translation allows normal memory accesses in the virtual machine
to execute without adding address translation overhead, after the shadow page tables are set
up. Because the translation look-aside buffer (TLB) on the processor caches direct virtual-tomachine mappings read from the shadow page tables, no additional overhead is added by the
VMM to access the memory.
The boxes represent pages, and the arrows show the different memory mappings.
The arrows from guest virtual memory to guest physical memory show the mapping
maintained by the page tables in the guest operating system. (The mapping from
virtual memory to linear memory for x86-architecture processors is not shown.)
The arrows from guest physical memory to machine memory show the mapping
maintained by the VMM.
The dashed arrows show the mapping from guest virtual memory to machine
memory in the shadow page tables also maintained by the VMM. The underlying
processor running the virtual machine uses the shadow page table mappings.
Because of the extra level of memory mapping introduced by virtualization, ESXi can
effectively manage memory across all virtual machines. Some of the physical memory of a
virtual machine might be mapped to shared pages or to pages that are unmapped, or
swapped out.
A host performs virtual memory management without the knowledge of the guest operating
system and without interfering with the guest operating systems own memory management
subsystem.
6.2.7 Performance Considerations
When you use hardware assistance, you eliminate the overhead for software memory
virtualization. In particular, hardware assistance eliminates the overhead required to keep
shadow page tables in synchronization with guest page tables. However, the TLB miss
latency when using hardware assistance is significantly higher. As a result, whether or not a
workload benefits by using hardware assistance primarily depends on the overhead the
memory virtualization causes when using software memory virtualization. If a workload
involves a small amount of page table activity (such as process creation, mapping the
memory, or context switches), software virtualization does not cause significant overhead.
Conversely, workloads with a large amount of page table activity are likely to benefit from
hardware assistance.
6.2.8 Overhead Memory on Virtual Machines
Virtual machines require a certain amount of available overhead memory to power on. You
should be aware of the amount of this overhead.
The following table lists the amount of overhead memory a virtual machine requires to
power on. After a virtual machine is running, the amount of overhead memory it uses might
differ from the amount listed in the table. The sample values were collected with VMX swap
enabled and hardware MMU enabled for the virtual machine. (VMX swap is enabled by
default.)
Note
The table provides a sample of overhead memory values and does not attempt to provide
information about all possible configurations. You can configure a virtual machine to have
up to 64 virtual CPUs, depending on the number of licensed CPUs on the host and the
number of CPUs that the guest operating system supports.
1 VCPU
2 VCPUs
4 VCPUs
8 VCPUs
256
20.29
24.28
32.23
48.16
1024
25.90
29.91
37.86
53.82
4096
48.64
52.72
60.67
76.78
16384
139.62
143.98
151.93
168.60
This approach ensures that a virtual machine from which idle memory is reclaimed can ramp
up quickly to its full share-based allocation when it starts using its memory more actively.
Memory activity is monitored to estimate the working set sizes for a default period of 60
seconds. To modify this default , adjust the Mem.SamplePeriod advanced setting. See Set
Advanced Host Attributes.
6.2.10 VMX Swap Files
Virtual machine executable (VMX) swap files allow the host to greatly reduce the amount of
overhead memory reserved for the VMX process.
Note
VMX swap files are not related to the swap to host cache feature or to regular host-level swap
files.
ESXi reserves memory per virtual machine for a variety of purposes. Memory for the needs
of certain components, such as the virtual machine monitor (VMM) and virtual devices, is
fully reserved when a virtual machine is powered on. However, some of the overhead
memory that is reserved for the VMX process can be swapped. The VMX swap feature
reduces the VMX memory reservation significantly (for example, from about 50MB or more
per virtual machine to about 10MB per virtual machine). This allows the remaining memory
Use the Mem.ShareScanTime and Mem.ShareScanGHz advanced settings to control the rate
at which the system scans memory to identify opportunities for sharing memory.
You can also disable sharing for individual virtual machines by setting the
sched.mem.pshare.enable option to FALSE (this option defaults to TRUE). See Set Advanced
Virtual Machine Attributes.
ESXi memory sharing runs as a background activity that scans for sharing opportunities over
time. The amount of memory saved varies over time. For a fairly constant workload, the
amount generally increases slowly until all sharing opportunities are exploited.
To determine the effectiveness of memory sharing for a given workload, try running the
workload, and use resxtop or esxtop to observe the actual savings. Find the information
in the PSHARE field of the interactive mode in the Memory page.
6.2.17 Memory Compression
ESXi provides a memory compression cache to improve virtual machine performance when
you use memory overcommitment. Memory compression is enabled by default. When a
host's memory becomes overcommitted, ESXi compresses virtual pages and stores them in
memory.
Because accessing compressed memory is faster than accessing memory that is swapped to
disk, memory compression in ESXi allows you to overcommit memory without significantly
hindering performance. When a virtual page needs to be swapped, ESXi first attempts to
compress the page. Pages that can be compressed to 2 KB or smaller are stored in the virtual
machine's compression cache, increasing the capacity of the host.
You can set the maximum size for the compression cache and disable memory compression
using the Advanced Settings dialog box in the vSphere Client.
6.2.18 Measuring and Differentiating Types of Memory Usage
The Performance tab of the vSphere Client displays a number of metrics that can be used to
analyze memory usage.
Some of these memory metrics measure guest physical memory while other metrics measure
machine memory. For instance, two types of memory usage that you can examine using
performance metrics are guest physical memory and machine memory. You measure guest
physical memory using the Memory Granted metric (for a virtual machine) or Memory
Shared (for a host). To measure machine memory, however, use Memory Consumed (for a
virtual machine) or Memory Shared Common (for a host). Understanding the conceptual
difference between these types of memory usage is important for knowing what these metrics
are measuring and how to interpret them.
The VMkernel maps guest physical memory to machine memory, but they are not always
mapped one-to-one. Multiple regions of guest physical memory might be mapped to the
same region of machine memory (in the case of memory sharing) or specific regions of guest
physical memory might not be mapped to machine memory (when the VMkernel swaps out
or balloons guest physical memory). In these situations, calculations of guest physical
memory usage and machine memory usage for an individual virtual machine or a host differ.
Consider the example in the following figure, which shows two virtual machines running on
a host. Each block represents 4 KB of memory and each color/letter represents a different set
of data on a block.
Memory Usage Example
The performance metrics for the virtual machines can be determined as follows:
To determine Memory Granted (the amount of guest physical memory that is mapped to
machine memory) for virtual machine 1, count the number of blocks in virtual machine 1's
guest physical memory that have arrows to machine memory and multiply by 4 KB. Since
there are five blocks with arrows, Memory Granted would be 20 KB.
Memory Consumed is the amount of machine memory allocated to the virtual machine,
accounting for savings from shared memory. First, count the number of blocks in machine
memory that have arrows from virtual machine 1's guest Measuring and Differentiating
Types of Memory Usage
The Performance tab of the vSphere Client displays a number of metrics that can be used to
analyze memory usage.
Some of these memory metrics measure guest physical memory while other metrics measure
machine memory. For instance, two types of memory usage that you can examine using
performance metrics are guest physical memory and machine memory. You measure guest
physical memory using the Memory Granted metric (for a virtual machine) or Memory
Shared (for a host). To measure machine memory, however, use Memory Consumed (for a
virtual machine) or Memory Shared Common (for a host). Understanding the conceptual
difference between these types of memory usage is important for knowing what these metrics
are measuring and how to interpret them.
The VMkernel maps guest physical memory to machine memory, but they are not always
mapped one-to-one. Multiple regions of guest physical memory might be mapped to the
same region of machine memory (in the case of memory sharing) or specific regions of guest
physical memory might not be mapped to machine memory (when the VMkernel swaps out
or balloons guest physical memory). In these situations, calculations of guest physical
memory usage and machine memory usage for an individual virtual machine or a host differ.
Consider the example in the following figure, which shows two virtual machines running on
a host. Each block represents 4 KB of memory and each color/letter represents a different set
of data on a block.
Memory Usage Example
The performance metrics for the virtual machines can be determined as follows:
The important difference between these two metrics is that Memory Granted counts the
number of blocks with arrows at the guest physical memory level and Memory Consumed
counts the number of blocks with arrows at the machine memory level. The number of blocks
differs between the two levels due to memory sharing and so Memory Granted and Memory
Consumed differ. This is not problematic and shows that memory is being saved through
sharing or other reclamation techniques.
A similar result is obtained when determining Memory Shared and Memory Shared
Common for the host.
Memory Shared for the host is the sum of each virtual machine's Memory Shared.
Calculate this by looking at each virtual machine's guest physical memory and
counting the number of blocks that have arrows to machine memory blocks that
themselves have more than one arrow pointing at them. There are six such blocks in
the example, so Memory Shared for the host is 24 KB.
Memory Shared Common is the amount of machine memory that is shared by virtual
machines. To determine this, look at the machine memory and count the number of
blocks that have more than one arrow pointing at them. There are three such blocks,
so Memory Shared Common is 12 KB.
Memory Shared is concerned with guest physical memory and looks at the origin of the
arrows. Memory Shared Common, however, deals with machine memory and looks at the
destination of the arrows.
The memory metrics that measure guest physical memory and machine memory might
appear contradictory. In fact, they are measuring different aspects of a virtual machine's
memory usage. By understanding the differences between these metrics, you can better
utilize them to diagnose performance issues.
Note
Storage I/O Control is enabled by default on Storage DRS-enabled datastore clusters.
Automated storage tiering is the ability of an array (or group of arrays) to migrate
LUNs/volumes or parts of LUNs/volumes to different types of storage media (SSD, FC, SAS,
SATA) based on user-set policies and current I/O patterns. No special certification is
required for arrays that do not have these automatic migration/tiering features, including
those that provide the ability to manually migrate data between different types of storage
media.
Caution
Storage I/O Control will not function correctly unless all datatores that share the same
spindles on the array have the same congestion threshold.
If you change the congestion threshold setting, set the value based on the following
considerations.
A higher value typically results in higher aggregate throughput and weaker isolation.
Throttling will not occur unless the overall average latency is higher than the threshold.
If throughput is more critical than latency, do not set the value too low. For example, for
Fibre Channel disks, a value below 20 ms could lower peak disk throughput. A very high
value (above 50 ms) might allow very high latency without any significant gain in overall
throughput.
A lower value will result in lower device latency and stronger virtual machine I/O
performance isolation. Stronger isolation means that the shares controls are enforced more
often. Lower device latency translates into lower I/O latency for the virtual machines with
the highest shares, at the cost of higher I/O latency experienced by the virtual machines with
fewer shares.
If latency is more important, a very low value (lower than 20 ms) will result in lower device
latency and better isolation among I/Os at the potential cost of a decrease in aggregate
datastore throughput.
6.5.1 Monitor Storage I/O Control Shares
Use the datastore Performance tab to monitor how Storage I/O Control handles the I/O
workloads of the virtual machines accessing a datastore based on their shares.
Datastore performance charts allow you to monitor the following information:
Procedure
1. Select the datastore in the vSphere Client inventory and click the Performance tab.
2. From the View drop-down menu, select Performance.
For more information, see the vSphere Monitoring and Performance documentation.
cluster itself represents the root resource pool. If you do not create child resource pools, only
the root resource pools exist.
In the following example, RP-QA is the parent resource pool for RP-QA-UI. RP-Marketing
and RP-QA are siblings. The three virtual machines immediately below RP-Marketing are
also siblings.
Parents, Children, and Siblings in Resource Pool Hierarchy
For each resource pool, you specify reservation, limit, shares, and whether the reservation
should be expandable. The resource pool resources are then available to child resource pools
and virtual machines.
You can create a child resource pool of any ESXi host, resource pool, or DRS cluster.
Note
If a host has been added to a cluster, you cannot create child resource pools of that host. If
the cluster is enabled for DRS, you can create child resource pools of the cluster.
When you create a child resource pool, you are prompted for resource pool attribute
information. The system uses admission control to make sure you cannot allocate resources
that are not available.
Prerequisites
The vSphere Client is connected to the vCenter Server system. If you connect the vSphere
Client directly to a host, you cannot create a resource pool.
Procedure
1. In the vSphere Client inventory, select a parent object for the resource pool (a host,
another resource pool, or a DRS cluster).
2. Select File > New > Resource Pool.
3. Type a name to identify the resource pool.
4. Specify how to allocate CPU and memory resources.
The CPU resources for your resource pool are the guaranteed physical resources the host
reserves for a resource pool. Normally, you accept the default and let the host handle
resource allocation.
Option
Description
Shares
Specify shares for this resource pool with respect to the parents total
resources. Sibling resource pools share resources according to their relative
share values bounded by the reservation and limit.
Select Low, Normal, or High to specify share values respectively in a 1:2:4
ratio.
Select Custom to give each virtual machine a specific number of shares,
which expresses a proportional weight.
Reservation
Expandable
Reservation
Limit
Specify the upper limit for this resource pools CPU or memory allocation.
You can usually accept the default (Unlimited).
To specify a limit, deselect the Unlimited check box
Expandable
(default)
The system checks whether the selected resource pool has sufficient
unreserved resources. If it does, the action can be performed. If it does not, a
message appears and the action cannot be performed.
The system considers the resources available in the selected resource pool
and its direct parent resource pool. If the parent resource pool also has the
Expandable Reservation option selected, it can borrow resources from its
parent resource pool. Borrowing resources occurs recursively from the
ancestors of the current resource pool as long as the Expandable Reservation
option is selected. Leaving this option selected offers more flexibility, but, at
the same time provides less protection. A child resource pool owner might
reserve more resources than you anticipate.
The system does not allow you to violate preconfigured Reservation or Limit settings. Each
time you reconfigure a resource pool or power on a virtual machine, the system validates all
parameters so all service-level guarantees can still be met.
6.8.1 Expandable Reservations Example 1
This example shows you how a resource pool with expandable reservations works.
Assume an administrator manages pool P, and defines two child resource pools, S1 and S2,
for two different users (or groups).
The administrator knows that users want to power on virtual machines with reservations,
but does not know how much each user will need to reserve. Making the reservations for S1
and S2 expandable allows the administrator to more flexibly share and inherit the common
reservation for pool P.
Without expandable reservations, the administrator needs to explicitly allocate S1 and S2 a
specific amount. Such specific allocations can be inflexible, especially in deep resource pool
hierarchies and can complicate setting reservations in the resource pool hierarchy.
Expandable reservations cause a loss of strict isolation. S1 can start using all of P's
reservation, so that no memory or CPU is directly available to S2.
Parent pool RP-MOM has a reservation of 6GHz and one running virtual machine
VM-M1 that reserves 1GHz.
You create a child resource pool RP-KID with a reservation of 2GHz and with
Expandable Reservation selected.
You add two virtual machines, VM-K1 and VM-K2, with reservations of 2GHz each
to the child resource pool and try to power them on.
VM-K1 can reserve the resources directly from RP-KID (which has 2GHz).
No local resources are available for VM-K2, so it borrows resources from the parent
resource pool, RP-MOM. RP-MOM has 6GHz minus 1GHz (reserved by the virtual
machine) minus 2GHz (reserved by RP-KID), which leaves 3GHz unreserved. With
3GHz available, you can power on the 2GHz virtual machine.
Admission Control with Expandable Resource Pools: Successful Power-On
Place the disks of all virtual machines on VMFS volumes that are accessible by source
and destination hosts.
Ensure the VMFS volume is sufficiently large to store all virtual disks for your virtual
machines.
Ensure all VMFS volumes on source and destination hosts use volume names, and all
virtual machines use those volume names for specifying the virtual disks.
Note
Virtual machine swap files also need to be on a VMFS accessible to source and destination
hosts (just like .vmdk virtual disk files). This requirement does not apply if all source and
destination hosts are ESX Server 3.5 or higher and using host-local swap. In that case,
vMotion with swap files on unshared storage is supported. Swap files are placed on a VMFS
by default, but administrators might override the file location using advanced virtual
machine configuration options.
Sometimes, processor vendors have introduced significant architectural changes within the
same processor family (such as 64-bit extensions and SSE3). VMware identifies these
exceptions if it cannot guarantee successful migration with vMotion.
vCenter Server provides features that help ensure that virtual machines migrated with
vMotion meet processor compatibility requirements. These features include:
Enhanced vMotion Compatibility (EVC) You can use EVC to help ensure vMotion
compatibility for the hosts in a cluster. EVC ensures that all hosts in a cluster present
the same CPU feature set to virtual machines, even if the actual CPUs on the hosts
differ. This prevents migrations with vMotion from failing due to incompatible CPUs.
Configure EVC from the Cluster Settings dialog box. The hosts in a cluster must meet
certain requirements for the cluster to use EVC. For information about EVC and EVC
requirements, see the vCenter Server and Host Management documentation.
CPU compatibility masks vCenter Server compares the CPU features available to a
virtual machine with the CPU features of the destination host to determine whether
to allow or disallow migrations with vMotion. By applying CPU compatibility masks
to individual virtual machines, you can hide certain CPU features from the virtual
machine and potentially prevent migrations with vMotion from failing due to
incompatible CPUs.
To be configured for vMotion, each host in the cluster must meet the following requirements:
vMotion does not support raw disks or migration of applications clustered using
Microsoft Cluster Service (MSCS).
vMotion requires a private Gigabit Ethernet migration network between all of the
vMotion enabled managed hosts. When vMotion is enabled on a managed host,
configure a unique network identity object for the managed host and connect it to the
private migration network.
Automation Level
Action
Manual
Partially Automated
Fully Automated
Note
If you have not enabled Enhanced vMotion Compatibility (EVC) for the cluster, fault tolerant
virtual machines are set to DRS disabled. They appear on this screen, but you cannot assign
an automation mode to them.
6.10.5 Add an Unmanaged Host to a Cluster
You can add an unmanaged host to a cluster. Such a host is not currently managed by the
same vCenter Server system as the cluster and it is not visible in the vSphere Client.
Procedure
1. Select the cluster to which to add the host and select Add Host from the right-click
menu.
2. Enter the host name, user name, and password, and click Next.
3. View the summary information and click Next.
4. Select what to do with the hosts virtual machines and resource pools.
Put this hosts virtual machines in the clusters root resource pool
vCenter Server removes all existing resource pools of the host and the virtual machines in the
hosts hierarchy are all attached to the root. Because share allocations are relative to a
resource pool, you might have to manually change a virtual machines shares after selecting
this option, which destroys the resource pool hierarchy.
Create a resource pool for this hosts virtual machines and resource pools
vCenter Server creates a top-level resource pool that becomes a direct child of the cluster and
adds all children of the host to that new resource pool. You can supply a name for that new
top-level resource pool. The default is Grafted from <host_name>.
Resource Pool Hierarchies When you remove a host from a cluster, the host retains only
the root resource pool, even if you used a DRS cluster and decided to graft the host resource
pool when you added the host to the cluster. In that case, the hierarchy remains with the
cluster. You can create a host-specific resource pool hierarchy.
Note
Ensure that you remove the host from the cluster by first placing it in maintenance mode. If
you instead disconnect the host before removing it from the cluster, the host retains the
resource pool that reflects the cluster hierarchy.
Virtual Machines A host must be in maintenance mode before you can remove it
from the cluster and for a host to enter maintenance mode all powered-on virtual
machines must be migrated off that host. When you request that a host enter
maintenance mode, you are also asked whether you want to migrate all the poweredoff virtual machines on that host to other hosts in the cluster.
Invalid Clusters When you remove a host from a cluster, the resources available for
the cluster decrease. If the cluster has enough resources to satisfy the reservations of
all virtual machines and resource pools in the cluster, the cluster adjusts resource
allocation to reflect the reduced amount of resources. If the cluster does not have
enough resources to satisfy the reservations of all resource pools, but there are
enough resources to satisfy the reservations for all virtual machines, an alarm is
issued and the cluster is marked yellow. DRS continues to run.
Server becomes available again, you might find that clusters have turned red or
yellow because cluster requirements are no longer met.
When considering cluster validity scenarios, you should understand these terms.
Reservation
Reservation
Used
Unreserved
A fixed, guaranteed allocation for the resource pool input by the user.
The sum of the reservation or reservation used (whichever is larger) for each
child resource pool, added recursively.
This nonnegative number differs according to resource pool type.
RP2 was created with a reservation of 4GHz. Two virtual machines of 1GHz and
2GHz are powered on (Reservation Used: 3GHz). 1GHz remains unreserved.
RP3 was created with a reservation of 3GHz. One virtual machine with 3GHz is
powered on. No resources for powering on additional virtual machines are available.
The following figure shows an example of a valid cluster with some resource pools (RP1 and
RP3) using reservation type Expandable.
Valid Cluster with Expandable Resource Pools
RP3 was created with a reservation of 5GHz. Two virtual machines of 3GHz and 2GHz are
powered on. Even though this resource pool is of type Expandable, no additional 2GHz
virtual machine can be powered on because the parents extra resources are already used by
RP1.
6.12.2 Overcommitted DRS Clusters
A cluster becomes overcommitted (yellow) when the tree of resource pools and virtual
machines is internally consistent but the cluster does not have the capacity to support all
resources reserved by the child resource pools.
There will always be enough resources to support all running virtual machines because,
when a host becomes unavailable, all its virtual machines become unavailable. A cluster
typically turns yellow when cluster capacity is suddenly reduced, for example, when a host in
the cluster becomes unavailable. VMware recommends that you leave adequate additional
cluster resources to avoid your cluster turning yellow.
Yellow Cluster
In this example:
A cluster with total resources of 12GHz coming from three hosts of 4GHz each.
Three resource pools reserving a total of 12GHz.
The total reservation used by the three resource pools combined is 12GHz
(4+5+3 GHz). That shows up as the Reserved Capacity in the cluster.
One of the 4GHz hosts becomes unavailable, so total resources reduce to 8GHz.
At the same time, VM4 (1GHz) and VM3 (3GHz), which were running on the host
that failed, are no longer running.
The cluster is now running virtual machines that require a total of 6GHz. The cluster
still has 8GHz available, which is sufficient to meet virtual machine requirements.
The resource pool reservations of 12GHz can no longer be met, so the cluster is marked as
yellow.
6.12.3 Invalid DRS Clusters
A cluster enabled for DRS becomes invalid (red) when the tree is no longer internally
consistent, that is, resource constraints are not observed.
The total amount of resources in the cluster does not affect whether the cluster is red. A
cluster can be red, even if enough resources exist at the root level, if there is an inconsistency
at a child level.
You can resolve a red DRS cluster problem either by powering off one or more virtual
machines, moving virtual machines to parts of the tree that have sufficient resources, or
editing the resource pool settings in the red part. Adding resources typically helps only when
you are in the yellow state.
A cluster can also turn red if you reconfigure a resource pool while a virtual machine is
failing over. A virtual machine that is failing over is disconnected and does not count toward
the reservation used by the parent resource pool. You might reduce the reservation of the
parent resource pool before the failover completes. After the failover is complete, the virtual
machine resources are again charged to the parent resource pool. If the pools usage becomes
larger than the new reservation, the cluster turns red.
If a user is able to start a virtual machine (in an unsupported way) with a reservation of
3GHz under resource pool 2, the cluster would become red, as shown in the following figure.
Red Cluster
6.13 DPM
Note
ESXi hosts cannot automatically be brought out of standby mode unless they are running in
a cluster managed by vCenter Server.
vSphere DPM can use one of three power management protocols to bring a host out of
standby mode: Intelligent Platform Management Interface (IPMI), Hewlett-Packard
Integrated Lights-Out (iLO), or Wake-On-LAN (WOL). Each protocol requires its own
hardware support and configuration. If a host does not support any of these protocols it
cannot be put into standby mode by vSphere DPM. If a host supports multiple protocols,
they are used in the following order: IPMI, iLO, WOL.
Your cluster must contain at least two ESX 3.5 (or ESX 3i version 3.5) or later hosts.
Each host's vMotion networking link must be working correctly. The vMotion
network should also be a single IP subnet, not multiple subnets separated by routers.
The vMotion NIC on each host must support WOL. To check for WOL support, first
determine the name of the physical network adapter corresponding to the VMkernel
port by selecting the host in the inventory panel of the vSphere Client, selecting the
Configuration tab, and clicking Networking. After you have this information, click on
Network Adapters and find the entry corresponding to the network adapter. The
Wake On LAN Supported column for the relevant adapter should show Yes.
To display the WOL-compatibility status for each NIC on a host, select the host in the
inventory panel of the vSphere Client, select the Configuration tab, and click Network
Adapters. The NIC must show Yes in the Wake On LAN Supported column.
The switch port that each WOL-supporting vMotion NIC is plugged into should be set
to auto negotiate the link speed, and not set to a fixed speed (for example, 1000
Mb/s). Many NICs support WOL only if they can switch to 100 Mb/s or less when the
host is powered off.
After you verify these prerequisites, test each ESXi host that is going to use WOL to support
vSphere DPM. When you test these hosts, ensure that the vSphere DPM feature is disabled
for the cluster.
Caution
Ensure that any host being added to a vSphere DPM cluster that uses WOL as a wake
protocol is tested and disabled from using power management if it fails the testing. If this is
not done, vSphere DPM might power off hosts that it subsequently cannot power back up.
Note
When you create a VM-Host affinity rule that is based on the licensing or hardware
requirements of the software running in your virtual machines, you are responsible for
ensuring that the groups are properly set up. The rule does not monitor the software running
in the virtual machines nor does it know what non-VMware licenses are in place on which
ESXi hosts.
If you create more than one VM-Host affinity rule, the rules are not ranked, but are applied
equally. Be aware that this has implications for how the rules interact. For example, a virtual
machine that belongs to two DRS groups, each of which belongs to a different required rule,
can run only on hosts that belong to both of the host DRS groups represented in the rules.
When you create a VM-Host affinity rule, its ability to function in relation to other rules is
not checked. So it is possible for you to create a rule that conflicts with the other rules you
are using. When two VM-Host affinity rules conflict, the older one takes precedence and the
newer rule is disabled. DRS only tries to satisfy enabled rules and disabled rules are ignored.
DRS, vSphere HA, and vSphere DPM never take any action that results in the violation of
required affinity rules (those where the virtual machine DRS group 'must run on' or 'must
not run on' the host DRS group). Accordingly, you should exercise caution when using this
type of rule because of its potential to adversely affect the functioning of the cluster. If
improperly used, required VM-Host affinity rules can fragment the cluster and inhibit the
proper functioning of DRS, vSphere HA, and vSphere DPM.
A number of cluster functions are not performed if doing so would violate a required affinity
rule.
DRS does not evacuate virtual machines to place a host in maintenance mode.
DRS does not place virtual machines for power-on or load balance virtual machines.
vSphere HA does not perform failovers.
vSphere DPM does not optimize power management by placing hosts into standby
mode.
To avoid these situations, exercise caution when creating more than one required affinity
rule or consider using VM-Host affinity rules that are preferential only (those where the
virtual machine DRS group 'should run on' or 'should not run on' the host DRS group).
Ensure that the number of hosts in the cluster with which each virtual machine is affined is
large enough that losing a host does not result in a lack of hosts on which the virtual machine
can run. Preferential rules can be violated to allow the proper functioning of DRS, vSphere
HA, and vSphere DPM.
Note
You can create an event-based alarm that is triggered when a virtual machine violates a VMHost affinity rule. In the vSphere Client, add a new alarm for the virtual machine and select
VM is violating VM-Host Affinity Rule as the event trigger. For more information about
creating and editing alarms, see the vSphere Monitoring and Performance documentation.
I/O Latency Storage DRS generates recommendations or performs migrations when the
90th percentile I/O latency measured over a day for the datastore is greater
than the threshold.
You can also set advanced options to further configure the aggressiveness level of Storage
DRS.
Space
utilization
difference
This threshold ensures that there is some minimum difference between the
space utilization of the source and the destination. For example, if the space
used on datastore A is 82% and datastore B is 79%, the difference is 3. If the
threshold is 5, Storage DRS will not make migration recommendations from
datastore A to datastore B.
I/O load
balancing
invocation
interval
I/O imbalance Lowering this value makes I/O load balancing less aggressive. Storage DRS
computes an I/O fairness metric between 0 and 1, which 1 being the fairest
threshold
distribution. I/O load balancing runs only if the computed metric is less
than 1 - (I/O imbalance threshold / 100).
A datastore cluster can contain a mix of datastores with different sizes and I/O capacities,
and can be from different arrays and vendors. However, the following types of datastores
cannot coexist in a datastore cluster.
NFS and VMFS datastores cannot be combined in the same datastore cluster.
Replicated datastores cannot be combined with non-replicated datastores in
the same Storage-DRS-enabled datastore cluster.
All hosts attached to the datastores in a datastore cluster must be ESXi 5.0 and later.
If datastores in the datastore cluster are connected to ESX/ESXi 4.x and earlier hosts,
Storage DRS does not run.
Datastores shared across multiple datacenters cannot be included in a datastore
cluster.
As a best practice, do not include datastores that have hardware acceleration enabled
in the same datastore cluster as datastores that do not have hardware acceleration
enabled. Datastores in a datastore cluster must be homogeneous to guarantee
hardware acceleration-supported behavior.
o
o
You can add to a datastore cluster any datastore that is mounted on a host in the vSphere
Client inventory, with the following exceptions:
All hosts attached to the datastore must be ESXi 5.0 and later.
The datastore cannot be in more than one datacenter in the same instance of the
vSphere Client.
When you remove a datastore from a datastore cluster, the datastore remains in the vSphere
Client inventory and is not unmounted from the host.
6.17.1 Place a Datastore in Maintenance Mode
If you need to take a datastore out of service, you can place the datastore in Storage DRS
maintenance mode.
Prerequisites
Storage DRS is enabled on the datastore cluster that contains the datastore that is entering
maintenance mode.
No CD-ROM image files are stored on the datastore.
There are at least two datastores in the datastore cluster.
Procedure
1. In the vSphere Client inventory, right-click a datastore in a datastore cluster and select
Enter SDRS Maintenance Mode.
A list of recommendations appears for datastore maintenance mode migration.
1. (Optional) On the Placement Recommendations tab, deselect any
recommendations you do not want to apply.
Note
The datastore cannot enter maintenance mode without evacuating all disks. If you deselect
recommendations, you must manually move the affected virtual machines.
Select IgnoreAffinityRulesForMaintenance.
In the Value column, type 1 to enable the option.
Type 0 to disable the option.
Click OK
Note
Anti-affinity rules do not apply to CD-ROM ISO image files that are stored on a datastore in
a datastore cluster, nor do they apply to swapfiles that are stored in user-defined locations.
If you move a virtual disk out of the datastore cluster, the affinity or anti-affinity rule no
longer applies to that disk.
When you move virtual disk files into a datastore cluster that has existing affinity and antiaffinity rules, the following behavior applies:
Datastore Cluster B has an intra-VM affinity rule. When you move a virtual disk out
of Datastore Cluster A and into Datastore Cluster B, any rule that applied to the
virtual disk for a given virtual machine in Datastore Cluster A no longer applies. The
virtual disk is now subject to the intra-VM affinity rule in Datastore Cluster B.
Datastore Cluster B has an inter-VM anti-affinity rule. When you move a virtual disk
out of Datastore Cluster A and into Datastore Cluster B, any rule that applied to the
virtual disk for a given virtual machine in Datastore Cluster A no longer applies. The
virtual disk is now subject to the inter-VM anti-affinity rule in Datastore Cluster B.
Datastore Cluster B has an intra-VM anti-affinity rule. When you move a virtual disk
out of Datastore Cluster A and into Datastore Cluster B, the intra-VM anti-affinity
rule does not apply to the virtual disk for a given virtual machine because the rule is
limited to only specified virtual disks in Datastore Cluster B.
Note
Storage DRS rules might prevent a datastore from entering maintenance mode. You can
choose to ignore Storage DRS rules for maintenance mode by enabling the Ignore Affinity
Rules for Maintenance option.
Storage DRS places the virtual machine's virtual disks according to the rule.
Storage DRS migrates the virtual disks using vMotion according to the rule, even if
the migration is for a mandatory reason such as putting a datastore in maintenance
mode.
If the virtual machine's virtual disk violates the rule, Storage DRS makes migration
recommendations to correct the error or reports the violation as a fault if it cannot
make a recommendation that will correct the error.
Procedure
1. In the vSphere Client inventory, right-click a datastore cluster and select Edit
Settings.
2. In the left pane of the Edit Datastore Cluster dialog box, select Rules.
3. Click Add.
4. Type a name for the rule.
5. From the Type menu, select VM anti-affinity.
6. Click Add.
7. Click Select Virtual Machine.
8. Select at least two virtual machines and click OK.
9. Click OK to save the rule.
The host must be running a version of ESXi that supports Storage vMotion.
The host must have write access to both the source datastore and the destination
datastore.
The host must have enough free memory resources to accommodate Storage vMotion.
The destination datastore must have sufficient disk space.
The destination datastore must not be in maintenance mode or entering maintenance
mode.
What is NUMA?
How ESXi NUMA Scheduling Works
VMware NUMA Optimization Algorithms and Settings
Resource Management in NUMA Architectures
Using Virtual NUMA
Specifying NUMA Controls
When a virtual machine moves to a new node, the ESXi host immediately begins to migrate
its memory in this fashion. It manages the rate to avoid overtaxing the system, particularly
when the virtual machine has little remote memory remaining or when the destination node
has little free memory available. The memory migration algorithm also ensures that the ESXi
host does not move memory needlessly if a virtual machine is moved to a new node for only a
short period.
When initial placement, dynamic rebalancing, and intelligent memory migration work in
conjunction, they ensure good memory performance on NUMA systems, even in the
presence of changing workloads. When a major workload change occurs, for instance when
new virtual machines are started, the system takes time to readjust, migrating virtual
machines and memory to new locations. After a short period, typically seconds or minutes,
the system completes its readjustments and reaches a steady state.
6.20.7 Transparent Page Sharing Optimized for NUMA
Many ESXi workloads present opportunities for sharing memory across virtual machines.
For example, several virtual machines might be running instances of the same guest
operating system, have the same applications or components loaded, or contain common
data. In such cases, ESXi systems use a proprietary transparent page-sharing technique to
securely eliminate redundant copies of memory pages. With memory sharing, a workload
running in virtual machines often consumes less memory than it would when running on
physical machines. As a result, higher levels of overcommitment can be supported efficiently.
Transparent page sharing for ESXi systems has also been optimized for use on NUMA
systems. On NUMA systems, pages are shared per-node, so each NUMA node has its own
local copy of heavily shared pages. When virtual machines use shared pages, they don't need
to access remote memory.
Note
This default behavior is the same in all previous versions of ESX and ESXi.
6.20.8
Resource Management in NUMA Architectures
You can perform resource management with different types of NUMA architecture.
With the proliferation of highly multicore systems, NUMA architectures are becoming more
popular as these architectures allow better performance scaling of memory intensive
workloads. All modern Intel and AMD systems have NUMA support built into the
processors. Additionally, there are traditional NUMA systems like the IBM Enterprise XArchitecture that extend Intel and AMD processors with NUMA behavior with specialized
chipset support.
Typically, you can use BIOS settings to enable and disable NUMA behavior. For example, in
AMD Opteron-based HP Proliant servers, NUMA can be disabled by enabling node
interleaving in the BIOS. If NUMA is enabled, the BIOS builds a system resource allocation
table (SRAT) which ESXi uses to generate the NUMA information used in optimizations. For
scheduling fairness, NUMA optimizations are not enabled for systems with too few cores per
NUMA node or too few cores overall. You can modify the numa.rebalancecorestotal and
numa.rebalancecoresnode options to change this behavior.
6.20.9
Using Virtual NUMA
vSphere 5.0 and later includes support for exposing virtual NUMA topology to guest
operating systems, which can improve performance by facilitating guest operating system
and application NUMA optimizations.
Virtual NUMA topology is available to hardware version 8 virtual machines and is enabled
by default when the number of virtual CPUs is greater than eight. You can also manually
influence virtual NUMA topology using advanced configuration options.
You can affect the virtual NUMA topology with two settings in the vSphere Client: number of
virtual sockets and number of cores per socket for a virtual machine. If the number of cores
per socket (cpuid.coresPerSocket) is greater than one, and the number of virtual cores in the
virtual machine is greater than 8, the virtual NUMA node size matches the virtual socket
size. If the number of cores per socket is less than or equal to one, virtual NUMA nodes are
created to match the topology of the first physical host where the virtual machine is powered
on.
When the number of virtual CPUs and the amount of memory used grow proportionately,
you can use the default values. For virtual machines that consume a disproportionally large
amount of memory, you can override the default values in one of the following ways:
Increase the number of virtual CPUs, even if this number of virtual CPUs is not used.
See Change the Number of Virtual CPUs.
Use advanced options to control virtual NUMA topology and its mapping over
physical NUMA topology. See Virtual NUMA Controls.
7 Security
VMware designed the virtualization layer, or VMkernel, to run virtual machines. It controls
the hardware that hosts use and schedules the allocation of hardware resources among the
virtual machines. Because the VMkernel is fully dedicated to supporting virtual machines
and is not used for other purposes, the interface to the VMkernel is strictly limited to the API
required to manage virtual machines.
ESXi provides additional VMkernel protection with the following features:
Memory
Hardening
Kernel Module Digital signing ensures the integrity and authenticity of modules, drivers and
applications as they are loaded by the VMkernel. Module signing allows ESXi
Integrity
to identify the providers of modules, drivers, or applications and whether
they are VMware-certified. VMware software and certain third-party drivers
are signed by VMware.
Trusted
vSphere uses Intel Trusted Platform Module/Trusted Execution Technology
Platform
(TPM/TXT) to provide remote attestation of the hypervisor image based on
Module (TPM) hardware root of trust. The hypervisor image comprises the following
elements:
ESXi software (hypervisor) in VIB (package) format
Third-party VIBs
Third-party drivers
To leverage this capability, your ESXi system must have TPM and TXT
enabled.
When TPM and TXT are enabled, ESXi measures the entire hypervisor stack
when the system boots and stores these measurements in the Platform
Configuration Registers (PCR) of the TPM. The measurements include the
VMkernel, kernel modules, drivers, native management applications that run
on ESXi, and any boot-time configuration options. All VIBs that are installed
on the system are measured.
Third-party solutions can use this feature to build a verifier that detects
tampering of the hypervisor image, by comparing the image with an image of
the expected known good values. vSphere does not provide a user interface to
view these measurements.
The measurements are exposed in a vSphere API. An event log is provided as
part of the API, as specified by the Trusted Computing Group (TCG)
Even a user with system administrator privileges on a virtual machines guest operating
system cannot breach this layer of isolation to access another virtual machine without
privileges explicitly granted by the ESXi system administrator. As a result of virtual machine
isolation, if a guest operating system running in a virtual machine fails, other virtual
machines on the same host continue to run. The guest operating system failure has no effect
on:
Each virtual machine is isolated from other virtual machines running on the same hardware.
Although virtual machines share physical resources such as CPU, memory, and I/O devices,
a guest operating system on an individual virtual machine cannot detect any device other
than the virtual devices made available to it.
Because the VMkernel mediates the physical resources and all physical hardware access
takes place through the VMkernel, virtual machines cannot circumvent this level of isolation.
Just as a physical machine communicates with other machines in a network through a
network card, a virtual machine communicates with other virtual machines running in the
same host through a virtual switch. Further, a virtual machine communicates with the
physical network, including virtual machines on other ESXi hosts, through a physical
network adapter.
If a virtual machine does not share a virtual switch with any other virtual machine, it
is completely isolated from virtual networks within the host.
If no physical network adapter is configured for a virtual machine, the virtual
machine is completely isolated from any physical networks.
If you use the same safeguards (firewalls, antivirus software, and so forth) to protect
a virtual machine from the network as you would for a physical machine, the virtual
machine is as secure as the physical machine.
You can further protect virtual machines by setting up resource reservations and limits on
the host. For example, through the detailed resource controls available in ESXi, you can
configure a virtual machine so that it always receives at least 10 percent of the hosts CPU
resources, but never more than 20 percent.
Resource reservations and limits protect virtual machines from performance degradation
that would result if another virtual machine consumed excessive shared hardware resources.
For example, if one of the virtual machines on a host is incapacitated by a denial-of-service
(DoS) attack, a resource limit on that machine prevents the attack from taking up so much of
the hardware resources that the other virtual machines are also affected. Similarly, a
resource reservation on each of the virtual machines ensures that, in the event of high
resource demands by the virtual machine targeted by the DoS attack, all the other virtual
machines still have enough resources to operate.
By default, ESXi imposes a form of resource reservation by applying a distribution algorithm
that divides the available host resources equally among the virtual machines while keeping a
certain percentage of resources for use by other system components. This default behavior
provides a degree of natural protection from DoS and distributed denial-of-service (DDoS)
attacks. You set specific resource reservations and limits on an individual basis to customize
the default behavior so that the distribution is not equal across the virtual machine
configuration.
VMware Security Resources on the Web
Topic
Resource
VMware securityhttp://www.vmware.com/security/
policy, up-to-
Resource
date security
alerts, security
downloads, and
focus
discussions of
security topics
http://www.vmware.com/support/policies/security_response.html
Corporate
VMware is committed to helping you maintain a secure environment.
security
Security issues are corrected in a timely manner. The VMware Security
response policy Response Policy states our commitment to resolve possible vulnerabilities
in our products.
http://www.vmware.com/support/policies/
VMware supports a variety of storage systems, software agents such as
backup agents, system management agents, and so forth. You can find lists
of agents, tools, and other software that supports ESXi by searching
http://www.vmware.com/vmtn/resources/ for ESXi compatibility guides.
Third-party
software support
policy
The industry offers more products and configurations than VMware can
Compliance and
security
standards, as
well as partner
solutions and in-http://www.vmware.com/go/compliance/
depth content
about
virtualization
and compliance
Information
about VMsafe
technology for
protection of
http://www.vmware.com/go/vmsafe/
Resource
virtual
machines,
including a list
of partner
solutions
If you have a firewall between your vCenter Server system and vCenter Server managed host,
open ports 443 and 903 in the firewall to allow data transfer to ESXi hosts from vCenter
Server .
For additional information on configuring the ports, see the firewall system administrator.
7.1.3 Connecting ESXi Hosts Through Firewalls
If you have a firewall between two ESXi hosts and you want to allow transactions between
the hosts or use vCenter Server to perform any source or target activities, such as vSphere
High Availability (vSphere HA) traffic, migration, cloning, or vMotion, you must configure a
connection through which the managed hosts can receive data.
To configure a connection for receiving data, open ports for traffic from services such as
vSphere High Availability, vMotion, and vSphere Fault Tolerance. See TCP and UDP Ports
for Management Access for a list of ports. Refer to the firewall system administrator for
additional information on configuring the ports.
7.1.4 TCP and UDP Ports for Management Access
vCenter Server, ESXi hosts, and other network components are accessed using
predetermined TCP and UDP ports. If you manage network components from outside a
firewall, you might be required to reconfigure the firewall to allow access on the appropriate
ports.
The table lists TCP and UDP ports, and the purpose and the type of each. Ports that are open
by default at installation time are indicated by (Default).
TCP and UDP Ports
Port
Purpose
Traffic Type
22
SSH Server
Incoming TCP
53
DNS Client
(Default)
Incoming and
outgoing UDP
68
DHCP Client
(Default)
Incoming and
outgoing UDP
Purpose
161
SNMP Server
(Default)
Traffic Type
Incoming UDP
NTP Client
Outgoing UDP
135
Used to join vCenter Virtual Appliance to sn Active Direcotry
(Default) domain
Incoming and
outgoing TCP
427
The CIM client uses the Service Location Protocol, version 2
(Default) (SLPv2) to find CIM servers.
Incoming and
outgoing UDP
HTTPS access
vCenter Server access to ESXi hosts
Default SSL Web port
vSphere Client access to vCenter Server
443
vSphere Client access to ESXi hosts
(Default)
WS-Management
Incoming TCP
The way you set up VLANs to secure parts of a network depends on factors such as the guest
operating system and the way your network equipment is configured.
VLANs are an effective means of controlling where and how widely data is transmitted
within the network. If an attacker gains access to the network, the attack is likely to be
limited to the VLAN that served as the entry point, lessening the risk to the network as a
whole.
VLANs provide protection only in that they control how data is routed and contained after it
passes through the switches and enters the network. You can use VLANs to help secure
Layer 2 of your network architecturethe data link layer. However, configuring VLANs
does not protect the physical layer of your network model or any of the other layers. Even if
you create VLANs, provide additional protection by securing your hardware (routers, hubs,
and so forth) and encrypting data transmissions.
VLANs are not a substitute for firewalls in your virtual machine configurations. Most
network configurations that include VLANs also include firewalls. If you include VLANs in
your virtual network, be sure that the firewalls that you install are VLAN-aware.
7.1.5.2 Properly Configure VLANs
Equipment misconfiguration and network hardware, firmware, or software defects can make
a VLAN susceptible to VLAN-hopping attacks.
VLAN hopping occurs when an attacker with authorized access to one VLAN creates packets
that trick physical switches into transmitting the packets to another VLAN that the attacker is
not authorized to access. Vulnerability to this type of attack usually results from a switch
being misconfigured for native VLAN operation, in which the switch can receive and
transmit untagged packets.
To help prevent VLAN hopping, keep your equipment up to date by installing hardware and
firmware updates as they become available. Also, follow your vendors best practice
guidelines when you configure your equipment.
VMware standard switches do not support the concept of a native VLAN. All data passed on
these switches is appropriately tagged. However, because other switches in the network might
be configured for native VLAN operation, VLANs configured with standard switches can
still be vulnerable to VLAN hopping.
If you plan to use VLANs to enforce network security, disable the native VLAN feature for
all switches unless you have a compelling reason to operate some of your VLANs in native
mode. If you must use native VLAN, see your switch vendors configuration guidelines for
this feature.
VMware standard switches provide safeguards against certain threats to VLAN security.
Because of the way that standard switches are designed, they protect VLANs against a
variety of attacks, many of which involve VLAN hopping.
Having this protection does not guarantee that your virtual machine configuration is
invulnerable to other types of attacks. For example, standard switches do not protect the
physical network against these attacks; they protect only the virtual network.
Standard switches and VLANs can protect against the following types of attacks.
MAC flooding
Doubleencapsulation
attacks
Spanning-tree
attacks
themselves as the root bridge. As the root bridge, the attacker can sniff the
contents of transmitted frames.
Random frame
attacks
VMware standard switches do not support STP and are not vulnerable to
this type of attack.
Involve sending large numbers of packets in which the source and
destination addresses stay the same, but in which fields are randomly
changed in length, type, or content. The goal of this attack is to force
packets to be mistakenly rerouted to a different VLAN.
VMware standard switches are not vulnerable to this type of attack.
Because new security threats develop over time, do not consider this an exhaustive list of
attacks. Regularly check VMware security resources on the Web to learn about security,
recent security alerts, and VMware security tactics.
the MAC address for the receiving network adapter in the destination MAC address field.
The receiving adapter accepts packets only when the destination MAC address in the packet
matches its own effective MAC address.
Upon creation, a network adapters effective MAC address and initial MAC address are the
same. The virtual machines operating system can alter the effective MAC address to another
value at any time. If an operating system changes the effective MAC address, its network
adapter receives network traffic destined for the new MAC address. The operating system can
send frames with an impersonated source MAC address at any time. This means an operating
system can stage malicious attacks on the devices in a network by impersonating a network
adapter that the receiving network authorizes.
You can use standard switch security profiles on hosts to protect against this type of attack by
setting three options. If you change any default settings for a port, you must modify the
security profile by editing standard switch settings in the vSphere Client.
7.2.1 MAC Address Changes
The setting for the MAC Address Changes option affects traffic that a virtual machine
receives.
When the option is set to Accept, ESXi accepts requests to change the effective MAC address
to other than the initial MAC address.
When the option is set to Reject, ESXi does not honor requests to change the effective MAC
address to anything other than the initial MAC address, which protects the host against MAC
impersonation. The port that the virtual adapter used to send the request is disabled and the
virtual adapter does not receive any more frames until it changes the effective MAC address
to match the initial MAC address. The guest operating system does not detect that the MAC
address change was not honored.
Note
The iSCSI initiator relies on being able to get MAC address changes from certain types of
storage. If you are using ESXi iSCSI and have iSCSI storage, set the MAC Address Changes
option to Accept.
In some situations, you might have a legitimate need for more than one adapter to have the
same MAC address on a networkfor example, if you are using Microsoft Network Load
Balancing in unicast mode. When Microsoft Network Load Balancing is used in the standard
multicast mode, adapters do not share MAC addresses.
MAC address changes settings affect traffic leaving a virtual machine. MAC address changes
will occur if the sender is permitted to make them, even if standard switches or a receiving
virtual machine does not permit MAC address
hanges.
7.2.2 Forged Transmissions
The setting for the Forged Transmits option affects traffic that is transmitted from a virtual
machine.
When the option is set to Accept, ESXi does not compare source and effective MAC
addresses.
To protect against MAC impersonation, you can set this option to Reject. If you do, the host
compares the source MAC address being transmitted by the operating system with the
effective MAC address for its adapter to see if they match. If the addresses do not match,
ESXi drops the packet.
The guest operating system does not detect that its virtual network adapter cannot send
packets by using the impersonated MAC address. The ESXi host intercepts any packets with
impersonated addresses before they are delivered, and the guest operating system might
assume that the packets are dropped.
7.2.3 Promiscuous Mode Operation
Promiscuous mode eliminates any reception filtering that the virtual network adapter would
perform so that the guest operating system receives all traffic observed on the wire. By
default, the virtual network adapter cannot operate in promiscuous mode.
Although promiscuous mode can be useful for tracking network activity, it is an insecure
mode of operation, because any adapter in promiscuous mode has access to the packets
regardless of whether some of the packets are received only by a particular network adapter.
This means that an administrator or root user within a virtual machine can potentially view
traffic destined for other guest or host operating systems.
Note
In some situations, you might have a legitimate reason to configure a standard switch to
operate in promiscuous mode (for example, if you are running network intrusion detection
software or a packet sniffer).
ability to protect data is its cipher strengththe number of bits in the encryption key. The
larger the number, the more secure the cipher.
To ensure the protection of the data transmitted to and from external network connections,
ESXi uses one of the strongest block ciphers available256-bit AES block encryption. ESXi
also uses 1024-bit RSA for key exchange. These encryption algorithms are the default for the
following connections.
vSphere Client connections to vCenter Server and to ESXi through the management
interface.
SDK connections to vCenter Server and to ESXi.
Management interface connections to virtual machines through the VMkernel.
SSH connections to ESXi through the management interface.
VMware does not support Version 1 SSH protocol and uses Version 2
protocol exclusively. Version 2 eliminates certain security problems present
in Version 1 and provides you with a safe way to communicate with the
management interface.
Improved
SSH supports only 256-bit and 128-bit AES ciphers for your connections.
cipher strength
These settings are designed to provide solid protection for the data you transmit to the
management interface through SSH. If this configuration is too restricted for your needs, you
can lower security parameters.
providers, and presents it to the outside world via standard APIs, the most common one
being WS-MAN.
Do not provide root credentials to remote applications to access the CIM interface. Instead,
create a service account specific to these applications and grant read-only access to CIM
information to any local account defined on the ESXi system, as well as any role defined in
vCenter Server.
Procedure
This role can be local to the host or centrally defined on vCenter Server, depending on how
the monitoring application works.
When a user logs into the host with the service account (for example, using the vSphere
Client), the user has only the privileges SystemManagement and CIMInteraction, or readonly access.
To improve security, restrict user access to the management interface and enforce access
security policies like setting up password restrictions.
The ESXi Shell has privileged access to certain parts of the host. Therefore, provide only
trusted users with ESXi Shell login access.
Also, strive to run only the essential processes, services, and agents such as virus checkers,
and virtual machine backups.
Whenever possible, use the vSphere Client or a third-party network management tool to
administer your ESXi hosts instead of working though the command-line interface as the
root user. Using the vSphere Client lets you limit the accounts with access to the ESXi Shell,
safely delegate responsibilities, and set up roles that prevent administrators and users from
using capabilities they do not need.
The host runs a variety of third-party packages to support management interfaces or tasks
that you must perform. VMware does not support upgrading these packages from anything
other than a VMware source. If you use a download or patch from another source, you might
compromise management interface security or functions. Regularly check third-party vendor
sites and the VMware knowledge base for security alerts.
In addition to implementing the firewall, risks to the hosts are mitigated using other
methods.
ESXi runs only services essential to managing its functions, and the distribution is
limited to the features required to run ESXi.
By default, all ports not specifically required for management access to the host are
closed. You must specifically open ports if you need additional services.
By default, weak ciphers are disabled and all communications from clients are
secured by SSL. The exact algorithms used for securing the channel depend on the
SSL handshake. Default certificates created on ESXi use SHA-1 with RSA encryption
as the signature algorithm.
The Tomcat Web service, used internally by ESXi to support access by Web clients,
has been modified to run only those functions required for administration and
monitoring by a Web client. As a result, ESXi is not vulnerable to the Tomcat security
issues reported in broader use.
VMware monitors all security alerts that could affect ESXi security and, if needed,
issues a security patch.
Insecure services such as FTP and Telnet are not installed, and the ports for these
services are closed by default. Because more secure services such as SSH and SFTP
are easily available, always avoid using these insecure services in favor of their safer
alternatives. If you must use insecure services and have implemented sufficient
protection for the host, you must explicitly open ports to support them.
Note
The firewall also allows Internet Control Message Protocol (ICMP) pings and
communication with DHCP and DNS (UDP only) clients.
Supported services and management agents that are required to operate the host are described
in a rule set configuration file in the ESXi firewall directory /etc/vmware/firewall/. The file
contains firewall rules and lists each rule's relationship with ports and protocols.
You cannot add a rule to the ESXi firewall unless you create and install a VIB that contains
the rule set configuration file. The VIB authoring tool is available to VMware partners.
Note
The behavior of the NFS Client rule set (nfsClient) is different from other rule sets. When the
NFS Client rule set is enabled, all outbound TCP ports are open for the destination hosts in
the list of allowed IP addresses. See NFS Client Rule Set Behavior for more information.
A numeric identifier for the service, if the configuration file contains more than one
service.
A unique identifier for the rule set, usually the name of the service.
For each rule, the file contains one or more port rules, each with a definition for
direction, protocol, port type, and port number or range of port numbers.
A flag indicating whether the service is enabled or disabled when the rule set is
applied.
An indication of whether the rule set is required and cannot be disabled.
Only users with the Administrator role can access the ESXi Shell. Users who are
in the Active Directory group ESX Admins are automatically assigned the
Administrator role. Any user with the Administrator role can execute system
commands (such as vmware -v) using the ESXi Shell.
To increase the security of your ESXi hosts, you can put them in lockdown mode.
When you enable lockdown mode, no users other than vpxuser have authentication
permissions, nor can they perform operations against the host directly. Lockdown mode
forces all operations to be performed through vCenter Server.
When a host is in lockdown mode, you cannot run vSphere CLI commands from an
administration server, from a script, or from vMA against the host. External software or
management tools might not be able to retrieve or modify information from the ESXi host.
Note
Users with the DCUI Access privilege are authorized to log in to the Direct Console User
Interface (DCUI) when lockdown mode is enabled. When you disable lockdown mode using
the DCUI, all users with the DCUI Access privilege are granted the Administrator role on the
host. You grant the DCUI Access privilege in Advanced Settings.
Enabling or disabling lockdown mode affects which types of users are authorized to access
host services, but it does not affect the availability of those services. In other words, if the
ESXi Shell, SSH, or Direct Console User Interface (DCUI) services are enabled, they will
continue to run whether or not the host is in lockdown mode.
You can enable lockdown mode using the Add Host wizard to add a host to vCenter Server,
using the vSphere Client to manage a host, or using the Direct Console User Interface
(DCUI).
Note
If you enable or disable lockdown mode using the Direct Console User Interface (DCUI),
permissions for users and groups on the host are discarded. To preserve these permissions,
you must enable and disable lockdown mode using the vSphere Client connected to vCenter
Server.
Lockdown mode is only available on ESXi hosts that have been added to vCenter Server.
This chapter includes the following topics:
Note
When you disable lockdown mode using the DCUI, all users with the DCUI Access privilege
are granted the Administrator role on the host.
Root users or users with the Administrator role on the host cannot log directly in to the host
with the DCUI if they have not been granted the DCUI Access privilege. If the
host is not managed by vCenter Server or if the host is unreachable, only DCUI Access users
can log into the DCUI and disable lockdown mode. If the DCUI service is stopped, you must
reinstall ESXi.
Different services are available to different types of users when the host is running in
lockdown mode, compared to when the host is running in normal mode. Nonroot users
cannot run system commands in the ESXi Shell.
Lockdown Mode Behavior
Service
Normal Mode
Lockdown Mode
vSphere
WebServices API
CIM Providers
Direct Console UI Users with Admin role on the host and users
(DCUI)
with the DCUI Access privilege
Normal Mode
Lockdown Mode
ESXi Shell
No users
SSH
No users
Caution
If you lose access to vCenter Server while running in Total Lockdown Mode, you must
reinstall ESXi to gain access to the host.
Default
Configuration
Recommended
Configuration
Total Lockdown
Configuration
Lockdown
Off
On
On
ESXi Shell
Off
Off
Off
SSH
Off
Off
Off
Direct Console UI
On
(DCUI)
On
Off
You cannot create ESXi users with the vSphere Web Client. You must log directly into
the host with the vSphere Client to create ESXi users.
ESXi 5.1 does not support local groups. However, Active Directory groups are
supported.
To prevent anonymous users such as root from accessing the host with the Direct Console
User Interface (DCUI) or ESXi Shell, remove the user's administrator privileges on the root
folder of the host. This applies to both local users and Active Directory users and groups.
Most inventory objects inherit permissions from a single parent object in the hierarchy. For
example, a datastore inherits permissions from either its parent datastore folder or parent
datacenter. Virtual machines inherit permissions from both the parent virtual machine
folder and the parent host, cluster, or resource pool simultaneously. To restrict a users
privileges on a virtual machine, you must set permissions on both the parent fo
7.9.1 Multiple Permission Settings
Objects might have multiple permissions, but only one permission for each user or group.
Permissions applied on a child object always override permissions that are applied on a
parent object. Virtual machine folders and resource pools are equivalent levels in the
hierarchy. If you assign propagating permissions to a user or group on a virtual machine's
folder and its resource pool, the user has the privileges propagated from the resource pool
and from the folder.
If multiple group permissions are defined on the same object and the user belongs to two or
more of those groups, two situations are possible:
If no permission is defined for the user on that object, the user is assigned the set of
privileges assigned to the groups for that object.
If a permission is defined for the user on that object, the user's permission takes
precedence over all group permissions.
User 1, who belongs to groups A and B, logs on. User 1 can both power on and take snapshots
of VM A and VM B.
Example 1: Inheritance of Multiple Permissions
User 1, who belongs to groups A and B, logs on. Because Role 2 is assigned at a lower point in
the hierarchy than Role 1, it overrides Role 1 on VM B. User 1 can power on VM A, but not
take snapshots. User 1 can take snapshots of VM B, but not power it on.
Example 2: Child Permissions Overriding Parent Permissions
User 1, who belongs to group A, logs on. The No Access role granted to User 1 on VM Folder
overrides the group permission. User 1 has no access to VM Folder or VMs A and B.
Example 3: User Permissions Overriding Group Permissions
Important
If you remove the access permissions for the root user, you must first create another
permission at the root level that has a different user assigned to the Administrator role.
Note
In vSphere 5.1, only the root user and no other user with administrator privileges is
permitted to add a host to vCenter Server.
Assigning the Administrator role to a different user helps you maintain security through
traceability. The vSphere Client logs all actions that the Administrator role user initiates as
events, providing you with an audit trail. If all administrators log in as the root user, you
cannot tell which administrator performed an action. If you create multiple permissions at
the root leveleach associated with a different useryou can track the actions of each
administrator.
Using default certificates might not comply with the security policy of your organization. If
you require a certificate from a trusted certificate authority, you can replace the default
certificate.
Note
If the host has Verify Certificates enabled, replacing the default certificate might cause
vCenter Server to stop managing the host. If the new certificate is not verifiable by vCenter
Server, you must reconnect the host using the vSphere Client.
ESXi supports only X.509 certificates to encrypt session information sent over SSL
connections between server and client components.
Note
Restart the host process after making any changes to host directories or authentication
mechanisms.
Do not set up certificates using pass phrases. ESXi does not support pass phrases, also
known as encrypted keys. If you set up a pass phrase, ESXi processes cannot start correctly.
You can configure the Web proxy so that it searches for certificates in a location other
than the default location. This capability proves useful for companies that prefer to
centralize their certificates on a single machine so that multiple hosts can use the
certificates.
Caution
If certificates are not stored locally on the hostfor example, if they are stored on an NFS
sharethe host cannot access those certificates if ESXi loses network connectivity. As a
result, a client connecting to the host cannot successfully participate in a secure SSL
handshake with the host.
To support encryption for user names, passwords, and packets, SSL is enabled by
default for vSphere Web services SDK connections. If you want to configure the these
connections so that they do not encrypt transmissions, disable SSL for your vSphere
Web Services SDK connection by switching the connection from HTTPS to HTTP.
Consider disabling SSL only if you created a fully trusted environment for these clients,
where firewalls are in place and transmissions to and from the host are fully isolated.
Disabling SSL can improve performance, because you avoid the overhead required to
perform encryption.
To protect against misuse of ESXi services, most internal ESXi services are accessible
only through port 443, the port used for HTTPS transmission. Port 443 acts as a
reverse proxy for ESXi. You can see a list of services on ESXi through an HTTP
welcome page, but you cannot directly access the Storage Adapters services without
proper authorization.
You can change this configuration so that individual services are directly accessible through
HTTP connections. Do not make this change unless you are using ESXi in a fully trusted
environment.
Any service running in a virtual machine provides the potential for attack. By disabling
unnecessary system components that are not necessary to support the application or service
running on the system, you reduce the number of components that can be attacked.
Virtual machines do not usually require as many services or functions as physical servers.
When you virtualize a system, evaluate whether a particular service or function is necessary.
Procedure
Disconnect unused physical devices, such as CD/DVD drives, floppy drives, and USB
adaptors.
See Removing Unnecessary Hardware Devices.
When you manually install guest operating systems and applications on a virtual machine,
you introduce a risk of misconfiguration. By using a template to capture a hardened base
operating system image with no applications installed, you can ensure that all virtual
machines are created with a known baseline level of security.
You can use these templates to create other, application-specific templates, or you can use the
application template to deploy virtual machines.
Procedure
1. Provide templates for virtual machine creation that contain hardened, patched, and
properly configured operating system deployments.
If possible, deploy applications in templates as well. Ensure that the applications do
not depend on information specific to the virtual machine to be deployed.
What to do next
You can convert a template to a virtual machine and back to a template in the vSphere Client,
which makes updating templates easy. For more information about templates, see the vSphere
Virtual Machine Administration documentation.
7.13.3 Prevent Virtual Machines from Taking Over Resources
When one virtual machine consumes so much of the host resources that other virtual
machines on the host cannot perform their intended functions, a Denial of Service (DoS)
might occur. To prevent a virtual machine from causing a DoS, use host resource
management features such as setting shares and limits to control the server resources
that a virtual machine consumes.
By default, all virtual machines on a host share resources equally.
Procedure
Ensure that unauthorized devices are not connected and remove any unneeded or
unused hardware devices.
Disable unnecessary virtual devices from within a virtual machine. An attacker with
access to a virtual machine can connect a disconnected CD-ROM drive and access
sensitive information on the media left in the drive, or disconnect a network adapter to
isolate the virtual machine from its network, resulting in a denial of service.
Ensure that no device is connected to a virtual machine if it is not required. Serial and
parallel ports are rarely used for virtual machines in a datacenter environment, and
CD/DVD drives are usually connected only temporarily during software installation.
For less commonly used devices that are not required, either the parameter should not
be
Maintain a supported operating system, database, and hardware for the vCenter
Server system. If vCenter Server is not running on a support operating system, it
might not run properly, making vCenter Server vulnerable to attacks.
Keep the vCenter Server system properly patched. By staying up-to-date with
operating system patches, the server is less vulnerable to attack.
Provide operating system protection on the vCenter Server host. Protection includes
antivirus and antimalware software.
For operating system and database compatibility information, see the vSphere Compatibility
Matrixes.
7.15.2 Best Practices for vCenter Server Privileges
Strictly control vCenter Server administrator privileges to increase security for the system.
Full administrative rights to vCenter Server should be removed from the local
Windows administrator account and granted to a special-purpose local vCenter
Server administrator account. Grant full vSphere administrative rights only to those
administrators who are required to have it. Do not grant this privilege to any group
whose membership is not strictly controlled.
Avoid allowing users to log in directly to the vCenter Server system. Allow only those
users who have legitimate tasks to perform to log into the system and ensure that
these events are audited.
Install vCenter Server using a service account instead of a Windows account. You can
use a service account or a Windows account to run vCenter Server. Using a service
account allows you to enable Windows authentication for SQL Server, which provides
more security. The service account must be an administrator on the local machine.
Check for privilege reassignment when you restart vCenter Server. If the user or user
group that is assigned the Administrator role on the root folder of the server cannot
be verified as a valid user or group, the Administrator privileges are removed and
assigned to the local Windows Administrators group.
Grant minimal privileges to the vCenter Server database user. The database user
requires only certain privileges specific to database access. In addition, some
privileges are required only for installation and upgrade. These can be removed after
the product is installed or upgraded.
Stagger the schedule for virus scans, particularly in deployments with a large number of
virtual machines. Performance of systems in your environment will degrade significantly if
you scan all virtual machines simultaneously.
Because software firewalls and antivirus software can be virtualization-intensive, you can
balance the need for these two security measures against virtual machine performance,
especially if you are confident that your virtual machines are in a fully trusted environment.
Configure persistent logging to a datastore. By default, the logs on ESXi hosts are
stored in the in-memory file system. Therefore, they are lost when you reboot the
host, and only 24 hours of log data is stored. When you enable persistent logging, you
have a dedicated record of server activity available for the host.
Remote logging to a central host allows you to gather log files onto a central host,
where you can monitor all hosts with a single tool. You can also do aggregate analysis
and searching of log data, which might reveal information about things like
coordinated attacks on multiple hosts.
Configure remote secure syslog on ESXi hosts using a remote command line such as
vCLI or PowerCLI, or using an API client.
Query the syslog configuration to make sure that a valid syslog server has been
configured, including the correct port.
This logging traffic between the Primary and Secondary VMs is unencrypted and contains
guest network and storage I/O data, as well as the memory contents of the guest operating
system. This traffic can include sensitive data such as passwords in plaintext. To avoid such
data being divulged, ensure that this network is secured, especially to avoid "man-in-themiddle" attacks. For example, use a private network for FT logging traffic.
The administrator password and user passwords that are included with the host
profile and answer file are MD5-encrypted. Any other passwords associated with host
profiles are in the clear.
Use the vSphere Authentication Service to set up Active Directory to avoid exposing
the Active Directory password. If you set up Active Directory using host profiles, the
passwords are not protected.
For more information about Auto Deploy, see the Auto Deploy information that is part of the
vSphere Installation and Setup documentation. For more information about host profiles
and answer files, see the vSphere Host Profiles documentation.
VMware Certified. VIBs that are VMware Certified are created, tested, and signed by
VMware.
VMware Accepted. VIBs that are created by a VMware partner, but tested and signed
by VMware.
Partner Supported. VIBs that are created, tested, and signed by a certified VMware
partner.
Community Supported. VIBs that have not been tested by VMware or a VMware
partner.
For more information about Image Builder, see the vSphere Installation and Setup
documentation.
Note
When the number of character classes is counted, the plug-in does not count uppercase
letters used as the first character in the password and numbers used as the last character of a
password.
To configure password complexity, you can change the default value of the following
parameters.
retry is the number of times a user is prompted for a new password if the password
candidate is not sufficiently strong.
N0 is the number of characters required for a password that uses characters from
only one character class. For example, the password contains only lowercase letters.
N1 is the number of characters required for a password that uses characters from two
character classes.
N2 is used for passphrases. ESXi requires three words for a passphrase. Each word in
the passphrase must be 8-40 characters long.
N3 is the number of characters required for a password that uses characters from
three character classes.
N4 is the number of characters required for a password that uses characters from all
four character classes.
match is the number of characters allowed in a string that is reused from the old
password. If the pam_passwdqc.so plug-in finds a reused string of this length or
longer, it disqualifies the string from the strength test and uses only the remaining
characters.
Setting any of these options to -1 directs the pam_passwdqc.so plug-in to ignore the
requirement.
Setting any of these options to disabled directs the pam_passwdqc.so plug-in to disqualify
passwords with the associated characteristic. The values used must be in descending order
except for -1 and disabled.
Note
The pam_passwdqc.so plug-in used in Linux provides more parameters than the parameters
supported for ESXi.
For more information on the pam_passwdqc.so plug-in, see your Linux documentation.
7.22.1 Change Default Password Complexity for the pam_passwdqc.so Plug-In
Configure the pam_passwdqc.so plug-in to determine the basic standards all passwords
must meet.
Procedure
1.
2.
3.
4.
5.
Important
To preclude the possibility that vCenter Server is locked out of the ESXi host, the password
aging policy must not be shorter than the interval that is set to automatically change the
vpxuser password.
Procedure
Default Location
Windows
Linux
/etc/vmware-vpx/vpxd.cfg
1. To change the password aging requirement, use the Advanced Settings dialog box in
the vSphere Web Client.
2. Browse to the vCenter Server system in the vSphere Web Client inventory.
3. Click the Manage tab and click Settings.
4. Select Advanced Settings and locate the
VirtualCenter.VimPasswordExpirationInDays parameter.
5. Restart vCenter Server.
To prevent a user other than the service account user from accessing the directory, change
the permissions on the directory so that only the vCenter Server service account is allowed to
access it. This restriction prevents you from collecting a complete support log when you issue
a vc-support script. The restriction also prevents the administrator from changing the
vCenter Server database password.
8 MSCS
Clustering Requirements
Component
Virtual SCSI adapter
Requirement
LSI Logic Parallel for Windows Server 2003
LSI Logic SAS for Windows Server 2008
Operating system
Virtual NIC
I/O timeout
Disk format
Number of nodes
NTP server
Storage Type
Clusters on
One Physical
Machine
(Cluster in a
Box)
Virtual disks
Yes
(recommended)
Pass-through RDM
(physical compatibility No
mode)
Clusters Across
Physical Machines
Clusters of Physical
and Virtual Machines
(Cluster Across
Boxes)
(Standby Host
Clustering)
No
No
Yes
(recommended)
Yes
Non-pass-through RDM
(virtual compatibility
mode)
Yes
Yes
No
Use of software iSCSI initiators within guest operating systems configured with MSCS, in any
configuration supported by Microsoft, is transparent to ESXi hosts and there is no need for
explicit support statements from VMware.
Note
Clusters across physical machines with non-pass-through RDM is supported only for
clustering with Windows Server 2003. It is not supported for clustering with Windows
Server 2008.
FCoE is supported in ESXi 5.1 Update 1. See KB 1037959 for more information.
8.1.3 MSCS and Booting from a SAN
You can put the boot disk of a virtual machine on a SAN-based VMFS volume.
Booting from a SAN is complex. Problems that you encounter in physical environments
extend to virtual environments. For general information about booting from a SAN, see the
vSphere Storage documentation.
Follow these guidelines when you place the boot disk of a virtual machine on a SAN-based
VMFS volume:
Consider the best practices for boot-from-SAN that Microsoft publishes in the
following knowledge base article: http://support.microsoft.com/kb/305547/en-us.
Use StorPort LSI Logic drivers instead of SCSIport drivers when running Microsoft
Cluster Service for Windows Server 2003 or 2008 guest operating systems.
Test clustered configurations in different failover scenarios before you put them into
production environments.
For more information, see Microsofts documentation for CCR or DAG on the Microsoft Web
site.
Failover clustering with Windows Server 2008 is not supported with virtual compatibility
mode (non-pass-through) RDMs.
8.3.1 Using vSphere DRS Groups and VM-Host Affinity Rules with MSCS
Virtual Machines
You can use the vSphere Client to set up two types of DRS groups: virtual machine DRS
groups, which contain at least one virtual machine, and host DRS groups, which contain at
least one host. A VM-Host affinity rule establishes an affinity (or anti-affinity) relationship
between a virtual machine DRS group and a host DRS group.
You must use VM-Host affinity rules because vSphere HA does not obey VM-VM affinity
rules. This means that if a host fails, vSphere HA might separate clustered virtual machines
that are meant to stay together, or vSphere HA might put clustered virtual machines that are
meant to stay apart on the same host. You can avoid this problem by setting up DRS groups
and using VM-Host affinity rules, which are obeyed by vSphere HA.
For a cluster of virtual machines on one physical host, all MSCS virtual machines must be in
the same virtual machine DRS group, linked to the same host DRS group with the affinity
rule "Must run on hosts in group."
For a cluster of virtual machines across physical hosts, each MSCS virtual machine must be
in a different virtual machine DRS group, linked to a different host DRS group with the
affinity rule "Must run on hosts in group."
Limit the number of hosts to two when you define host DRS group rules for a
cluster of virtual machines on one physical host. (This does not apply to clusters
of virtual machines across physical hosts.) Since vSphere HA does not obey VMVM affinity rules, virtual machines in the configuration could be spread across
hosts during a vSphere HA recovery from host failure if more than two hosts
are included in a host DRS group rule.
Each type of clustered disk has its own requirements, depending on whether it is in a singlehost cluster or multihost cluster.
Requirements for Clustered Disks
Component
Clustered virtual disk
(.vmdk)
Single-Host Clustering
Multihost Clustering
SCSI bus sharing mode
Not supported.
must be set to virtual.
Device type must be set
Device type must be set to virtual
to virtual compatibility
compatibility mode for cluster across
mode.
boxes, but not for standby host clustering
or cluster across boxes on Windows Sever
Clustered disks, virtual SCSI bus sharing mode
2008.
compatibility mode
must be set to virtual
(non-pass-through
mode.
SCSI bus sharing mode must be set to
RDM)
physical.
A single, shared RDM
mapping file for each
Requires a single, shared RDM mapping
clustered disk is
file for each clustered disk.
required.
Device type must be set to Physical
compatibility mode during hard disk
Clustered disks,
creation.
physical compatibility
Not supported.
mode (pass-through
SCSI bus sharing mode must be set to
RDM)
physical (the default).
Component
All types
Single-Host Clustering
Multihost Clustering
A single, shared RDM mapping file for
each clustered disk is required.
All clustered nodes must use the same target ID (on the virtual SCSI
adapter) for the same clustered disk.
A separate virtual adapter must be used for clustered disks.
The only disks that you should not create with the Thick Provision option are
RDM files (both physical and virtual compatibility mode).
Use Windows Server 2003 SP1 and SP2 (32 bit), Windows Server 2003 SP1 and
SP2 (64 bit), Windows Server 2008 SP2 (32 bit), Windows Server 2008 SP2 (64
bit), or Windows Server 2008 SP1 R2 (32 bit), Windows Server 2008 SP1 R2 (64
bit)
For Windows Server 2003 SP1 and SP2, use only two cluster nodes.
For Windows Server 2008 SP2 and above, you can use up to five cluster nodes.
Windows
ESXi
configurati If you must overcommit memory, the swap file must be local, not on the SAN.
on
ESXi 5.0 uses a different technique to determine if Raw Device Mapped (RDM)
LUNs are used for MSCS cluster devices, by introducing a configuration flag to
Caution
Do not change, move, or delete these files without instructions from a VMware Technical
Support Representative.
Usage
Description
.vmx
vmname.vmx
.vmxf
vmname.vmxf
.vmdk
vmname.vmdk
Usage
Description
vmname-flat.vmdk
flat.vmdk
.nvram
vmname.nvram or nvram
.vmsd
vmname.vmsd
.vmsn
vmname.vmsn
.vswp
vmname.vswp
.vmss
vmname.vmss
.log
vmware.log
-#.log
Each virtual device performs the same function for the virtual machine as hardware on a
physical computer does.
A virtual machine might be running in any of several locations, such as ESXi hosts,
datacenters, clusters, or resource pools. Many of the options and resources that you configure
have dependencies on and relationships with these objects.
Every virtual machine has CPU, memory, and disk resources. CPU virtualization emphasizes
performance and runs directly on the processor whenever possible. The underlying physical
resources are used whenever possible. The virtualization layer runs instructions only as
needed to make virtual machines operate as if they were running directly on a physical
machine.
All recent operating systems provide support for virtual memory, allowing software to use
more memory than the machine physically has. Similarly, the ESXi hypervisor provides
support for overcommitting virtual machine memory, where the amount of guest memory
configured for all virtual machines might be larger than the amount of the host's physical
memory.
You can add virtual disks and add more space to existing disks, even when the virtual
machine is running. You can also change the device node and allocate shares of disk
bandwidth to the virtual machine.
VMware virtual machines have the following options:
Supported Features for Virtual Machine Compatibility
and later
ESXi 5.0
and later
ESX/ESXi 4.x
and later
ESX/ESXi
3.5 and
later
Hardware version
1035264
1035264
261120
65532
64
32
64
32
ESXi 5.1
Feature
Use the thin provisioned format. At first, a thin provisioned disk uses only as
much datastore space as the disk initially needs. If the thin disk needs more
space later, it can grow to the maximum capacity allocated to it.
The guest operating system customization feature in vCenter Server and VMware vCenter
Server Appliance uses the functions of the Sysprep tool. Verify that your vCenter Server or
VMware vCenter Server Appliance system meets the following requirements before you
customize your virtual machines Windows guest operating systems:
Install the Microsoft Sysprep tool. Microsoft includes the system tool set on the
installation CD-ROM discs for Windows 2000, Windows XP, and Windows 2003.
The Sysprep tool is built into the Windows Vista and Windows 2008 operating
systems.
The correct versions of the Sysprep tool is installed for each guest operating system
that you want to customize.
The password for the local administrator account on the virtual machines is set to
blank ("").
If you are using the VMware vCenter Server Application, you must have access to the
VMware vCenter Server Appliance Web console.
Note
Customization operations will fail if the correct version of the Sysprep tool is not found.
9.2.1 Install the Microsoft Sysprep Tool from a Microsoft Web Site
You can download and install the Microsoft Sysprep tool from the Microsoft Web site.
Prerequisites
Verify that you download the correct version for the guest operating system to customize.
Microsoft has a different version of Sysprep for each release and service pack of Windows.
You must use the version of Sysprep specific to the operating system that you are deploying.
The vCenter Server installer creates a Sysprep directory in ALLUSERSPROFILE. The
ALLUSERSPROFILE location is usually \Documents And Settings\All Users\. The vpxd.cfg
file is also in this location. On Windows 2008, the file location is
C:\ProgramData\VMware\VMware VirtualCenter\sysprep\.
Procedure
1. Download the Sysprep files from the Microsoft Download Center and save them to
your local system.
2. Open and expand the .cab file.
The contents of the .cab file vary, depending on the operating system.
3. Extract the files to the appropriate directory for your guest operating system.
The following Sysprep support directories are created during the vCenter Server
installation:
C:\ALLUSERSPROFILE\Application Data\Vmware\VMware VirtualCenter\syspr
ep
...\1.1\
...\2k\
...\xp\
...\svr2003\
...\xp-64\
...\svr2003-64\
9.2.2 Install the Microsoft Sysprep Tool from the Windows Operating System
CD
You can install the Microsoft Sysprep tool from a CD.
The vCenter Server installer creates a Sysprep directory in ALLUSERSPROFILE. The
1. Insert the Windows operating system CD into the CD-ROM drive, often the D: drive.
2. Locate the DEPLOY.CAB file in the \Support\Tools directory on the CD.
3. Open and expand the DEPLOY.CAB file.
The contents of the .cab file vary, depending on the operating system.
4. Extract the files to the directory appropriate for your guest operating system.
The following Sysprep support directories are created during the vCenter Server
installation:
C:\ALLUSERSPROFILE\Application Data\Vmware\VMware VirtualCenter\syspr
ep
...\1.1\
...\2k\
...\xp\
...\svr2003\
...\xp-64\
...\svr2003-64\
7. Repeat this procedure to extract Sysprep files for each of the Windows guest
operating systems that you plan to customize using vCenter Server.
9.2.3 Install the Microsoft Sysprep Tool for VMware vCenter Server Appliance
After you download and install the Microsoft Sysprep tool from the Microsoft Web site, you
can use the VMware vCenter Server Appliance Web console to upload the files to the
appliance.
Prerequisites
Verify that you download the correct version for the guest operating system to customize.
Microsoft has a different version of Sysprep for each release and service pack of Windows.
You must use the version of Sysprep specific to the operating system that you are deploying.
When you upload the files to vCenter Server Appliance, the contents of the CAB file for the
Sysprep Tool version that you downloaded are saved in /etc/vmware-vpx/sysprep/OS. For
example, /etc/vmware-vpx/sysprep/2k or /etc/vmware-vpx/sysprep/xp.
Procedure
1. Download the Sysprep files from the Microsoft Download Center and save them to
your local system.
2. Log in to the VMware vCenter Server Appliance Web console and click the vCenter
Server Summary tab.
3. In the Utilities panel, click the Sysprep Files Upload button.
4. Select a Windows platform directory, and browse to the file.
5. Click Open.
The file is uploaded to the VCenter Server Appliance.
6. Click Close.
You can customize a new virtual machine with a supported Windows guest operating system
when you clone an existing virtual machine.
This virtual machine (hardware version 9) is compatible with ESXi 5.1 and
later.
This virtual machine (hardware version 8) is compatible with ESXi 5.0 and
5.1.
ESX/ESXi 4.x This virtual machine (hardware version 7) is compatible with ESX/ ESXi
and later
4.x, ESXi 5.0, and ESXi 5.1.
ESX/ESXi 3.5 This virtual machine (hardware version 4) is compatible with ESX/ESXi 3.5.
and later
ESX/ESXi 4.x, and ESXi 5.1. It is also compatible with VMware Server 1.0
You can accept the default compatibility or select a different setting. It is not always
necessary to select the latest ESXi host version. Selecting an earlier version can provide
greater flexibility and is useful in the following situations:
Action
Virtual
machine
Select a virtual machine in the inventory and click the Summary tab.
Host
Cluster
Select a cluster in the inventory, click the Manage tab, and in the
Configuration section, click General.
Datacenter
You can change the default compatibility or upgrade the virtual machine compatibility.
9.3.2 Supported Features for Virtual Machine Compatibility
Feature
ESX/ESXi 3.5
and later
Hardware version
1035264
1035264
261120
65532
64
32
64
32
9.4 Change CPU Hot Plug Settings in the vSphere Web Client
The CPU hot plug option lets you add CPU resources for a virtual machine while the machine
is turned on.
The following conditions apply:
For best results, use virtual machines that are compatible with ESXi 5.0 and later.
Hot-adding multicore virtual CPUs is supported only with virtual machine that are
compatible with ESXi 5.0 and later.
Not all guest operating systems support CPU hot add. You can disable these settings
if the guest is not supported.
To use the CPU hot-add feature with virtual machines that are compatible with ESXi
4.x and later, set the Number of cores per socket to 1.
Adding CPU resources to a running virtual machine with CPU hot plug enabled
disconnects and reconnects all USB passthrough devices connected to that virtual
machine.
Prerequisites
Verify that the virtual machine is running under the following conditions:
VMware Tools is installed. This condition is required for hot plug functionality with
Linux guest operating systems.
The virtual machine has a guest operating system that supports CPU hot plug.
The virtual machine compatibility is ESX/ESXi 4.x or later.
The virtual machine is turned off.
9.4.1 Change CPU Identification Mask Settings in the vSphere Web Client
CPU identification (CPU ID) masks control the CPU features visible to the virtual machine's
guest operating system. Masking or hiding CPU features can make a virtual machine widely
available to ESXi hosts for migration. vCenter Server compares the CPU features available to
a virtual machine with the CPU features of the destination host to determine whether to
allow or disallow migration with vMotion.
For example, masking the AMD No eXecute (NX) and the Intel eXecute Disable (XD) bits
prevents the virtual machine from using these features, but provides compatibility that
allows you to migrate virtual machines to ESXi hosts that do not include this capability.
When the NX/XD bit is visible to the guest operating system, the virtual machine can use
this feature,
but you can migrate the virtual machine only to hosts on which the feature is enabled.
Caution
Changing the CPU compatibility masks can result in an unsupported configuration. Do not
manually change the CPU compatibility masks unless instructed to do so by VMware
Support or a VMware Knowledge base article.
Prerequisites
Description
Increases vMotion compatibility.
Hiding the NX/XD flag increases vMotion compatibility between hosts,
but might disable certain CPU security features.
3. Click OK.
9.4.2 Expose VMware Hardware Assisted Virtualization in the vSphere Web
Client
You can expose full CPU virtualization to the guest operating system so that applications that
require hardware virtualization can run on virtual machines without binary translation or
paravirtualization.
Prerequisites
Verify that the virtual machine compatibility is ESXi 5.1 and later.
Intel Nehalem Generation (Xeon Core i7) or later processors or AMD Opteron
Generation 3 (Greyhound) or later processors.
Verify that Intel VT-x or AMD-V is enabled in the BIOS so that hardware assisted
virtualization is possible.
Required Privileges: Virtual machine.Configuration.Settings set on the vCenter
Server system.
Procedure
The Manage tab refreshes, and the Nested Hypervisor CPU option shows Enabled.
9.4.3 Allocate Memory Resources in the vSphere Web Client
You can change the amount of memory resources allocated to a virtual machine by using the
shares, reservations, and limits settings. The host determines the appropriate amount of
physical RAM to allocate to virtual machines based on these settings. You can assign a high
or low shares value to a virtual machine, depending on its load and status.
The following user-defined settings affect the memory resource allocation of a virtual
machine.
Limit Places a limit on the consumption of memory for a virtual machine. This value is
expressed in megabytes.
Reservation Specifies the guaranteed minimum allocation for a virtual machine. The
reservation is expressed in megabytes. If the reservation cannot be met, the
virtual machine will not turn on.
Shares Each virtual machine is granted a number of memory shares. The more shares a
virtual machine has, the greater share of host memory it receives. Shares represent a
relative metric for allocating memory capacity. For more information about share
values, see the vSphere Resource Management documentation.
You cannot assign a reservation to a virtual machine that is larger than its configured
memory. If you give a virtual machine a large reservation and reduce its configured memory
size, the reservation is reduced to match the new configured memory size.
9.4.4 Network Adapter Types
When you configure a virtual machine, you can add network adapters (NICs) and specify the
adapter type.
The type of network adapters that are available depend on the following factors:
The virtual machine version, which depends on what host created it or most recently
updated it.
Whether the virtual machine has been updated to the latest version for the current
host.
The guest operating system.
Emulated version of the Intel 82545EM Gigabit Ethernet NIC, with drivers
available in most newer guest operating systems, including Windows XP
and later and Linux versions 2.4.19 and later.
Flexible
Vlance
VMXNET
VMXNET 2
(Enhanced)
Description
Dependent
Independent Persistent
Independent Nonpersistent
Changes to disks in nonpersistent mode are discarded when you turn off
or reset the virtual machine. With nonpersistent mode, you can restart
the virtual machine with a virtual disk in the same state every time.
Changes to the disk are written to and read from a redo log file that is
deleted when you turn off or reset the virtual machine.
9.5.2 Use Disk Shares to Prioritize Virtual Machines in the vSphere Web Client
You can change the disk resources for a virtual machine. If multiple virtual machines access
the same VMFS datastore and the same logical unit number (LUN), use disk shares to
prioritize the disk accesses from the virtual machines. Disk shares distinguish high-priority
from low-priority virtual machines.
You can allocate the host disk's I/O bandwidth to the virtual hard disks of a virtual machine.
Disk I/O is a host-centric resource so you cannot pool it across a cluster.
Shares is a value that represents the relative metric for controlling disk bandwidth to all
virtual machines. The values are compared to the sum of all shares of all virtual machines on
the server.
Disk shares are relevant only within a given host. The shares assigned to virtual machines on
one host have no effect on virtual machines on other hosts.
You can select an IOP limit, which sets an upper bound for storage resources that are
allocated to a virtual machine. IOPs are the number of I/O operations per second.
Procedure
By default, the SCSI controller is assigned to virtual device node (z:7), so that device node is
unavailable for hard disks or SCSI devices.
9.6.1 About VMware Paravirtual SCSI Controllers
VMware Paravirtual SCSI controllers are high performance storage controllers that can
result in greater throughput and lower CPU use. These controllers are best suited for high
performance storage environments.
VMware Paravirtual SCSI controllers are available for virtual machines with ESXi 4.x and
later compatibility. Disks on such controllers might not experience optimal performance
gains if they have snapshots or if memory on the ESXi host is over committed. This behavior
does not mitigate the overall performance gain of using VMware Paravirtual SCSI controllers
as compared to other SCSI controller options.
If you have virtual machines with VMware Paravirtual SCSI controllers, those virtual
machines cannot be part of an MSCS cluster.
For platform support for VMware Paravirtual SCSI controllers, see the VMware
Compatibility Guide.
9.6.2 Add a PCI Device in the vSphere Web Client
vSphere DirectPath I/O allows a guest operating system on a virtual machine to directly
access physical PCI and PCIe devices connected to a host. This action gives you direct access
to devices such as high-performance graphics or sound cards. You can connect each virtual
machine to up to six PCI devices.
You configure PCI devices on the host to make them available for passthrough to a virtual
machine. See the vCenter Server and Host Management documentation.
When PCI vSphere DirectPath I/O devices are available to a virtual machine, you cannot
suspend, migrate with vMotion, or take or restore Snapshots of such virtual machines.
Prerequisites
To use DirectPath, verify that the host has Intel Virtualization Technology for
Directed I/O (VT-d) or AMD I/O Virtualization Technology (IOMMU) enabled in the
BIOS.
Verify that the PCI devices are connected to the host and marked as available for
passthrough.
Verify that the virtual machine is compatible with ESXi 4.x and later.
You can add up to 20 USB devices to a virtual machine. This is the maximum
number of devices supported for simultaneous connection to one virtual
machine. The maximum number of USB devices supported on a single ESXi
host for simultaneous connection to one or more virtual machines is also 20.
For a list of supported USB devices, see the VMware knowledge base article at
http://kb.vmware.com/kb/1021345.
9.6.5 Configuring USB Devices for vMotion
With USB passthrough from a host to a virtual machine, you can migrate a virtual machine
to another ESXi host in the same datacenter and maintain the USB passthrough device
connections to the original host.
If a virtual machine has USB devices attached that pass through to an ESXi host, you can
migrate that virtual machine with the devices attached.
For a successful migration, review the following conditions:
You must configure all USB passthrough devices connected to a virtual machine for
vMotion. If one or more devices is not configured for vMotion, the migration cannot
proceed. For troubleshooting details, see the vSphere Troubleshooting
documentation.
When you migrate a virtual machine with attached USB devices away from the host
to which the devices are connected, the devices remain connected to the virtual
machine. However, if you suspend or power off the virtual machine, the USB devices
are disconnected and cannot reconnect when the virtual machine is resumed. The
device connections can be restored only if you move the virtual machine back to the
host to which the devices are attached.
If you resume a suspended virtual machine that has a Linux guest operating system,
the resume process might mount the USB devices at a different location on the file
system.
If a host with attached USB devices resides in a DRS cluster with distributed power
management (DPM) enabled, disable DPM for that host. Otherwise DPM might turn
off the host with the attached device. This action disconnects the device from the
virtual machine because the virtual machine migrated to another host.
Remote USB devices require that the hosts be able to communicate over the
management network following migration with vMotion, so the source and
destination management network IP address families must match. You cannot
migrate a virtual machine from a host that is registered to vCenter Server with an
IPv4 address to a host that is registered with an IPv6 address.
9.6.6 Add a USB Controller to a Virtual Machine in the vSphere Web Client
USB controllers are available to add to virtual machines to support USB passthrough from an
ESXi host or from a client computer to a virtual machine.
You can add two USB controllers to a virtual machine. The xHCI controller, available for
Linux guest operating systems only, supports USB 3.0 superspeed, 2.0, and 1.1 devices. The
EHCI+UHCI controller supports USB 2.0 and 1.1 devices.
The conditions for adding a controller vary, depending on the device version, the type of
passthrough (host or client computer), and the guest operating system.
Supported for
Supported for
Supported USB
Passthrough from ESXi Passthrough from Client
Device Version
Host to VM
Computer to VM
Yes
Yes
xHCI
For Mac OS X systems, the EHCI+UHCI controller is enabled by default and is required for
USB mouse and keyboard access.
For virtual machines with Linux guests, you can add one or both controllers, but 3.0
superspeed devices are not supported for passthrough from an ESXi host to a virtual
machine. You cannot add two controllers of the same type.
For USB passthrough from an ESXi host to a virtual machine, the USB arbitrator can
monitor a maximum of 15 USB controllers. If your system includes controllers that exceed
the 15 controller limit and you connect USB devices to them, the devices are not available to
the virtual machine.
Prerequisites
ESXi hosts must have USB controller hardware and modules that support USB 2.0
and 1.1 devices present.
Client computers must have USB controller hardware and modules that support USB
3.0, 2.0, and 1.1 devices present.
To use the xHCI controller on a Linux guest, ensure that the Linux kernel version is
2.6.35 or later.
Verify that the virtual machine is powered on.
Required Privilege (ESXi host passthrough): Virtual Machine.Configuration.Add or
Remove Device
A USB device is available to only one powered-on virtual machine at a time. When a virtual
machine connects to a device, that device is no longer available to other virtual machines or
to the client computer. When you disconnect the device from the virtual machine or shut the
virtual machine down, the device returns to the client computer and becomes available to
other virtual machines that the client computer manages.
For example, when you connect a USB mass storage device to a virtual machine, it is
removed from the client computer and does not appear as a drive with a removable device.
When you disconnect the device from the virtual machine, it reconnects to the client
computer's operating system and is listed as a removable device.
9.6.8 USB 3.0 Device Limitations
USB 3.0 devices have the following requirements and limitations:
The virtual machine that you connect the USB 3.0 device to must be configured with
an xHCI controller and have a Linux guest operating system with a 2.6.35 or later
kernel.
You can connect only one USB 3.0 device operating at superspeed to a virtual
machine at a time.
USB 3.0 devices are available only for passthrough from a client computer to a virtual
machine. They are not available for passthrough from an ESXi host to a virtual
machine.
HBA port among multiple virtual ports, each with unique identifiers. This capability lets you
control virtual machine access to LUNs on a per-virtual machine basis.
Each virtual port is identified by a pair of world wide names (WWNs): a world wide port
name (WWPN) and a world wide node name (WWNN). These WWNs are assigned by
vCenter Server.
For detailed information on how to configure NPIV for a virtual machine, see vSphere
Storage.
NPIV must be enabled on the SAN switch. Contact the switch vendor for information
about enabling NPIV on their devices.
NPIV is supported only for virtual machines with RDM disks. Virtual machines with
regular virtual disks continue to use the WWNs of the hosts physical HBAs.
The physical HBAs on the ESXi host must have access to a LUN using its WWNs in
order for any virtual machines on that host to have access to that LUN using their
NPIV WWNs. Ensure that access is provided to both the host and the virtual
machines.
The physical HBAs on the ESXi host must support NPIV. If the physical HBAs do not
support NPIV, the virtual machines running on that host will fall back to using the
WWNs of the hosts physical HBAs for LUN access.
Each virtual machine can have up to 4 virtual ports. NPIV-enabled virtual machines
are assigned exactly 4 NPIV-related WWNs, which are used to communicate with
physical HBAs through virtual ports. Therefore, virtual machines can utilize up to 4
physical HBAs for NPIV purposes.
Prerequisites
To edit the virtual machines WWNs, power off the virtual machine.
Verify that the virtual machine has a datastore containing a LUN that is available to
the host.
Version 7
Version 4
ESXi 5.0
ESX/ESXi
4.x
Not
supported
Create, edit,
VirtualCenter Server 2.x and later
run
Version 3 virtual machines are not supported on ESXi 5.0 hosts. To make full use of these
virtual machines, upgrade the virtual hardware.
Note
Virtual machine hardware version 4 might be listed as VM3 in documentation for earlier
versions of ESX/ESXi.
Note
The vApp metadata resides in the vCenter Server's database, so a vApp can be distributed
across multiple ESXi hosts. This information can be lost if the vCenter Server database is
cleared or if a standalone ESXi host that contains a vApp is removed from vCenter Server.
You should back up vApps to an OVF package to avoid losing any metadata.
vApp metadata for virtual machines within vApps do not follow the snapshots semantics for
virtual machine configuration. So, vApp properties that are deleted, modified, or defined
after a snapshot is taken remain intact (deleted, modified, or defined) after the virtual
machine reverts to that snapshot or any prior snapshots.
You can use VMware Studio to automate the creation of ready-to-deploy vApps with prepopulated application software and operating systems. VMware Studio adds a network agent
to the guest so that vApps bootstrap with minimal effort. Configuration parameters specified
for vApps appear as OVF properties in the vCenter Server deployment wizard. For
information about VMware Studio and for download, see the VMware Studio developer page
on the VMware web site.
You can allocate CPU and memory resources for the new vApp using shares, reservations,
and limits.
Procedure
Option
Description
Shares
CPU shares for this vApp with respect to the parents total. Sibling vApps share
resources according to their relative share values bounded by the reservation
and limit. Select Low, Normal, or High, which specify share values respectively
in a 1:2:4 ratio. Select Custom to give each vApp a specific number of shares,
which express a proportional weight.
Upper limit for this vApp's CPU allocation. Select Unlimited to specify no upper
limit.
Description
Shares
Memory shares for this vApp with respect to the parents total. Sibling vApps
share resources according to their relative share values bounded by the
reservation and limit. Select Low, Normal, or High, which specify share values
respectively in a 1:2:4 ratio. Select Custom to give each vApp a specific number
of shares, which express a proportional weight.
Upper limit for this vApp's memory allocation. Select Unlimited to specify no
upper limit.
When a solution's state changes, the solutions manager updates the ESX Agent Manager's
summary status and state. Administrators use this status to track whether the goal state is
reached.
The agency's health status is indicated by a specific color:
Red. The solution must intervene for the ESX Agent Manager to proceed. For
example, if a virtual machine agent is powered off manually on a compute resource
and the ESX Agent Manager does not attempt to power on the agent. The ESX Agent
Manager reports this action to the solution. The solution alerts the administrator to
power on the agent.
Yellow. The ESX Agent Manager is actively working to reach a goal state. The goal
state can be enabled, disabled, or uninstalled. For example, when a solution is
registered, its status is yellow until the ESX Agent Manager deploys the solutions
agents to all the specified compute resources. A solution does not need to intervene
when the ESX Agent Manager reports its ESX Agent Manager health status as yellow.
Green. A solution and all its agents reached the goal state.
Importing OVF templates. Receive a callback when OVF templates with a vService
dependancy of a certain type is imported.
Exporting OVF templates. Inserts OVF sections when a virtual machine is exported.
OVF environment generation. Inserts OVF sections into the OVF environment at the
power-on instance.
The vService Provider tab in the solution manager provides details for each vCenter
extension. This information allows you to monitor vService providers and list the virtual
machines or vApps to which they are bound.
9.10.1 Install the Client Integration Plug-In in the vSphere Web Client
The Client Integration Plug-in provides access to a virtual machine's console in the vSphere
Web Client, and provides access to other vSphere infrastructure tasks.
You use the Client Integration Plug-in to deploy OVF or OVA templates and transfer files
with the datastore browser. You can also use the Client Integration Plug-in to connect virtual
devices that reside on a client computer to a virtual machine.
You install the Client Integration Plug-in only once to connect virtual devices to virtual
machines that you access through an instance of the vSphere Web Client. You must restart
the browser after you install the plug-in.
If you install the Client Integration Plug-in from an Internet Explorer browser, you must first
disable Protected Mode. Internet Explorer identifies the Client Integration Plug-in as being on
the Internet instead of on the local intranet. In such cases, the plug-in does not install
correctly because Protected Mode is enabled for the Internet.
The Client Integration Plug-in also enables you to log in to the vSphere Web Client using
Windows session credentials.
For information about supported browsers and operating systems, see the vSphere Installation
and Setup documentation.
With snapshots, you can preserve a baseline before diverging a virtual machine in the
snapshot tree.
The Snapshot Manager in the vSphere Web Client and the vSphere Client provide several
operations for creating and managing virtual machine snapshots and snapshot trees. These
operations let you create snapshots, restore any snapshot in the snapshot hierarchy, delete
snapshots, and more. You can create extensive snapshot trees that you can use to save the
virtual machine state at any specific time and restore the virtual machine state later. Each
branch in a snapshot tree can have up to 32 snapshots.
A snapshot preserves the following information:
Virtual machine settings. The virtual machine directory, which includes disks that
were added or changed after you took the snapshot.
Power state. The virtual machine can be powered on, powered off, or suspended.
Disk state. State of all the virtual machine's virtual disks.
(Optional) Memory state. The contents of the virtual machine's memory.
The first virtual machine snapshot that you create is the base parent snapshot.
The parent snapshot is the most recently saved version of the current state of
the virtual machine. Taking a snapshot creates a delta disk file for each disk
attached to the virtual machine and optionally, a memory file. The delta disk
files and memory file are stored with the base .vmdk file. The parent snapshot
is always the snapshot that appears immediately above the You are here icon
in the Snapshot Manager. If you revert or restore a snapshot, that snapshot
becomes the parent of the You are here current state.
Note
The parent snapshot is not always the snapshot that you took most recently.
Child
Snapshots
A snapshot that is taken of the same virtual machine after the parent
snapshot. Each child constitutes delta files for each attached virtual disk, and
optionally a memory file that points from the present state of the virtual disk
(You are here). Each child snapshot's delta files merge with each previous
child snapshot until reaching the parent disks. A child disk can later be a
parent disk for future child disks.
The relationship of parent and child snapshots can change if you have multiple branches in
the snapshot tree. A parent snapshot can have more than one child. Many snapshots have no
children.
Important
Do not manually manipulate individual child disks or any of the snapshot configuration files
because doing so can compromise the snapshot tree and result in data loss. This restriction
includes disk resizing and making modifications to the base parent disk using vmkfstools.
virtual machine. These states are saved to files that reside with the virtual machine's base
files.
9.11.3.1 Snapshot Files
A snapshot consists of files that are stored on a supported storage device. A Take Snapshot
operation creates .vmdk, -delta.vmdk, .vmsd, and .vmsn files. By default, the first and all
delta disks are stored with the base .vmdk file. The .vmsd and .vmsn files are stored in the
virtual machine directory.
A .vmdk file to which the guest operating
system can write. The delta disk represents
the difference between the current state of the
virtual disk and the state that existed at the
time that the previous snapshot was taken.
When you take a snapshot, the state of the
virtual disk is preserved, which prevents the
guest operating system from writing to it, and
a delta or child disk is created.
A delta disk has two files, including a
descriptor file that is small and contains
information about the virtual disk, such as
geometry and child-parent relationship
information, and a corresponding file that
contains the raw data.
Note
If you are looking at a datastore with the
Datastore Browser in the vSphere Client, you
see only one entry to represent both files.
Memory file
A Take Snapshot operation creates .vmdk, -delta.vmdk, vmsd, and vmsn files.
File
Description
vmname-number.vmdk and
vmname-number-delta.vmdk
vmname.vmsd
vmname.Snapshotnumber.vmsn
Note
A .vmsn file is created each time you take a snapshot,
regardless of the memory selection. A .vmsn file without
memory is much smaller than one with memory.
9.11.4 Snapshot Limitations
Snapshots can affect virtual machine performance and do not support some disk types or
virtual machines configured with bus sharing. Snapshots are useful as short-term solutions
for capturing point-in-time virtual machine states and are not appropriate for long-term
virtual machine backups.
VMware does not support snapshots of raw disks, RDM physical mode disks, or guest
operating systems that use an iSCSI initiator in the guest.
Virtual machines with independent disks must be powered off before you take a
snapshot. Snapshots of powered-on or suspended virtual machines with independent
disks are not supported.
Snapshots are not supported with PCI vSphere Direct Path I/O devices.
VMware does not support snapshots of virtual machines configured for bus sharing.
If you require bus sharing, consider running backup software in your guest operating
system as an alternative solution. If your virtual machine currently has snapshots
that prevent you from configuring bus sharing, delete (consolidate) the snapshots.
Snapshots provide a point-in-time image of the disk that backup solutions can use,
but Snapshots are not meant to be a robust method of backup and recovery. If the
files containing a virtual machine are lost, its snapshot files are also lost. Also, large
numbers of snapshots are difficult to manage, consume large amounts of disk space,
and are not protected in the case of hardware failure.
Snapshots can negatively affect the performance of a virtual machine. Performance
degradation is based on how long the snapshot or snapshot tree is in place, the depth
of the tree, and how much the virtual machine and its guest operating system have
changed from the time you took the snapshot. Also, you might see a delay in the
amount of time it takes the virtual machine to power-on. Do not run production
virtual machines from snapshots on a permanent basis.
Snapshot tree
Represents the current and active state of the virtual machine. The You
are here icon is always selected and visible when you open the Snapshot
Manager.
You can select the You are here state to see how much space the node is
using. Go to, Delete, and Delete all are disabled for the You are here state.
Go to, Delete,
and Delete All
Snapshot options.
Details
Shows the snapshot name and description, the date you created the
snapashot, and the disk space.
The Console shows the power state of the virtual machine when a
snapshot was taken. The Name, Description, and Created text boxes are
blank if you do not select a snapshot.
Navigation
Procedure
Option
Independent Persistent
Independent Nonpersistent
Description
Click OK.
The current disk and memory states are discarded, and the virtual machine reverts to
the disk and memory states of the parent snapshot.
Existing snapshots are not removed. You can restore those snapshots at any time.
If the snapshot includes the memory state, the virtual machine will be in the same
power state as when you created the snapshot.
Virtual Machine Power State After Restoring a Snapshot
Virtual Machine State When
Parent Snapshot Is Taken
Virtual machines running certain kinds of workloads can take several minutes to resume
responsiveness after reverting from a snapshot.
Revert to Snapshot
Note
vApp metadata for virtual machines in vApps does not follow the snapshot semantics for
virtual machine configuration. vApp properties that are deleted, modified, or defined after a
snapshot is taken remain intact (deleted, modified, or defined) after the virtual machine
reverts to that snapshot or any previous snapshots.