Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
esxcfg-
There are a new set of command line tools in ESX 3.x which all start with
"esxcfg-". These tools are used to configure each part of the ESX 3.x
configuration. For example, esxcfg-firewall is used to manage the service console
firewall while the esxcfg-nic is used to manage the physical Ethernet adapters
present in the server.
Watch out for vicfg- commands also. If you are using the RCLI tools for
managing ESX 3i, then the esxcfg- tools are now prefixed with vicfg- although
the esxcfg- prefix still works.
esxcfg-advcfg
The esxcfg-advcfg command is interesting as there is not a huge amount of help
about this command. However, we can figure out that it is meant to do advanced
configuration and we can figure out some settings that can be made. The -g
switch is used to "get" settings; the -s switch is used to "set" settings.
The question is, how much is configurable? To figure out what is configurable, we
recommend that you look in the directory /proc/vmware/config which you will
find in the service console command line and then you will see the following
directories
BufferCache
Cpu
Disk
FileSystem
Irq
LVM
Mem
Migrate
Misc
Net
NFS
Numa
Scsi
User
VMFS3
From these directories and the files within, you can work out the paths to be
supplied to the esxcfg-advcfg command as parameters. Alternatively, you could
also use the command
esxcfg-info –o
We often see this tool used to make configuration changes relating to storage. For
example, below, you can see we are checking to see if we are creating virtual
disks in "eager zero" format by default, whether we will discover non-contiguous
numbered LUNs, the maximum LUN number addressed,the SCSI conflict retry
count and finally the logical volume manager (LVM) setting for resignaturing
VMFS volumes.
In this last example, we are again setting a parameter related to storage. This
parameter limits the number of outstanding disk request for each VM. This is
intended to equalise the disk access between virtual machines.
esxcfg-firewall
The service console in ESX 3.x has a firewall enabled by default. The network
packet filtering found in Red Hat Linux is called iptables. As the management
of iptables is not entirely straightforward, the esxcfg-firewall command
makes things a load easier. The firewall rules are stored in /etc/vmware/esx.conf,
but we don't go editing this file, we use this command to ensure it is locked while
we make our edits. If you are very interested in the iptables commands used
behind the scenes, then you can inspect the log file/var/log/vmware/esxcfg-
firewall.log
We use the esxcfg-firewall command to view and configure the firewall rules. The
most popular switch will be the -q switch to query the firewall for its current
settings.
<output>
The -s switch will allow you to enable or disable network services that may
traverse the firewall successfully. The list of known services are shown below -
very case sensitive!....
nfsClient
ftpServer
ntpClient
dellom
nisClient
vncServer
tmpLicenseClient
swISCSIClient
CIMHttpsServer
sshClient
snmpd
tmpAAMClient
vpxHeartbeats
smbClient
hpim
tmpHostVmdbServer
tmpHostdSOAPServer
ftpClient
sshServer
ibmdirector
CIMHttpServer
telnetClient
If we need to open a TCP or UDP port that is not described by a defined friendly
name like "sshClient", then we can explicitly open that port with the -o switch.
The service console firewall is bidirectional and so when opening a port you must
also specify direction of incoming or outgoing. Equally, we can close an explicit
port with the -c switch.
In the following example, we are opening a unique port which we are calling
"MySQLclient". If we wanted to close a port that we had already opened, we
would use the -cswitch.
The service names such as sshClient and smbClient are defined in the
file /etc/vmware/firewall/services.xml. It is strongly suggested that this file
is not manually edited as changes are unlikely to survive host patch updates. A
much better approach for defining services is to add a new XML file, for example
the guys over at Veeam very helpfully have already created one for you so you
can enable their FastSCP tool
- http://www.veeam.com/download/fastscp/FastSCP.xml. See more under the guide
entry forservices.xml.
esxcfg-module
This command is used to view and set options for start-up on the VMkernel
modules (drivers). When this command is used with the list option, it produces an
output similar tovmkload_mod -list
esxcfg-rescan
This command is used to perform a rescan of a host bus adapter (HBA).
Specifically it scans a named vmkernel hba device, i.e. a vmhba. This command
does a similar job tovmkfstools -rescan.
esxcfg-upgrade
esxcfg-upgrade -h --help
-g --convert-grub
-f --convert-fstab
-r --upgrade-pre-vmkernel
-o --upgrade-post-vmkernel
esxcfg-vswitch
This command is one of the most useful commands in the service console. This
command allows you to list, add, modify or delete virtual Ethernet switches on an
ESX host. The simplest option with this command is the -l option to list the virtual
switches and portgroups defined on the host.
If you are having problems with your ESX server after an in-place upgrade, this
tool is invaluable in resolving the problems with service console networking. The
output of this command is initially a little intimidating. It is best to keep in mind
the network topology:
Service Console IP Interface (vswif0) ---- connected to ----> Service Console Port
on vSwitch ----- up-linked to ----> vmnic
In following screenshot taken from the VI Client, we can see this ESX host has 2
connections to vSwitch0, the service console connection a VMkernel port
connection.
If we wish to view the same information at the service console command line, we
would use the esxcfg-vswitch command with the "-l" switch to list the defined
virtual switches.
Notice that the number of ports on the virtual switch is 64 on the newly created
switch. The original virtual switch has only 32. This difference arises between
creating the switch in the VI Client or the command line. Notice also that the used
port count doesn't immediately make sense. Each VM consumes a port, each
vmnic consumes a port and the uplink itself consumes a port. Anyway, if you are
like me and you can never remember which case of the letter "a" to use when
adding a virtual switch, then use theesxcfg-vswitch command with the --
add switch when creating a new switch like this:
Now if we want to add a portgroup to the new virtual switch we have created, we
can use the esxcfg-vswitch -A command. It does not matter whether you are
creating a service console port, a VM port group or a VMkernel port when creating
a port group; the way we create the connection to the virtual switch always starts
out the same in the command line. Only after creating the port group do we then
specify if it is to be anything other than a VM port group. In the following
commands, we add a new portgroup called "Production" on the virtual switch
vSwitch1.
Alternatively you could use the following command to add a port group to a
virtual switch.
This alternative switch of using --ad-pg I think is clearer for understanding what
the command is doing. The --add-pg option can clearly be seen to add a
portgroup to a virtual switch, and again is simpler to understand than just “-A”.
The portgroup name in our example is called “Production”, but it can be what you
want. We recommend adoption of a standard across all your virtual infrastructure.
I have seen some clients align their portgroup names with the IP subnets, so you
could have a portgroup called something like “192.168.1.0 subnet”.
Although we have now created a new virtual switch and have created a VM port
group on it, the virtual switch itself does not have any uplinks. Remember that
when we bind a physical network adapter to a virtual switch we are uplinking a
vmnic to the switch and the switch then "owns" that adapter, i.e. it is not
available to be used by any other virtual switches. We perform the uplink by
using the esxcfg-vswitch command with the -L switch for link.
So in one simple command we have linked the physical network adapter vmnic1
to our new virtual Ethernet switch vSwitch1. If we then realised we had used the
wrong physical adapter, we can just as easily unlink with -U. In the next example,
we swap the uplinked vmnic1 for an alternative adapter vmnic2
If we wish to do VLAN tagging in the virtual switch (VST), then we can assign a
VLAN ID to a port group using the -v switch to this command. All traffic passing
through this portgroup will now be tagged (IEEE 802.1q) with the VLAN ID
specified as a numeric parameter after the -v switch. This must match the VLAN
ID of the network defined in the physical switch topology in the range 1 through
4094. The physical switch port that the traffic uplinks through from ESX will also
need to be configured to accept q-tagged traffic for that VLAN. In Cisco
terminology this is a trunk port, in HP ProCurve terminology this is a tagged port.
If you wanted to do VLAN tagging in the guest operating system itself - called
Virtual Guest Tagging (VGT), then you can set the VLAN ID of the port group to
4095, which allows tagged traffic from the guest to pass through the portgroup.
As of ESX 3.5, VMware added Cisco Discovery Protocol (CDP) support for virtual
switches. We can view CDP information of the current neighbour of the physical
NIC. In the VI Client, we can see this by clicking on the icon to the right side of
the vmnic in the network view of the ESX host.
To display the CDP configuration setting for a virtual switch, we use the lowercase
b switch, where we will find which of the four CDP modes it is in: disable, listen,
advertise or both.
We can change the CDP mode with the -B (uppercase) option. Here we are
changing virtual switch called vSwitch0 to support both advertise and listen.
esxcfg-auth
Configures the service console user authentication options including NIS, LDAP,
Kerberos and Active Directory. In the following command, we are configuring
authentication for the Active Directory domain called taupoconsulting.com
You can also use this tool to set a password policy for service console user
accounts.
In the above example, your service console user account password would expire
after 90 days, you would get a warning message after 75 and once changed, you
would have to keep that password for a minimum period of 30 days.
esxcfg-info
Produces an enormous amount of information about the state ESX host, often this
tool is the one tool that can tell you what is really going on and not what is in
some configuration file. If you run this command with no parameters, then you
really need to pipe this to a file for closer examination! Over time as newer
releases of ESX server are released, less information will be available in the proc
nodes (the /proc/vmware directory structure), so the sooner we can get used to
examining the current running configuration of ESX using this command, the
better off we will be.
In this first example, we will run the command with no switches and pipe the
result into a file esxinfo-28-07-2008.txt (we like putting in the date of
operation into the filenames of dumped files so we don't lose track!) and we are
then viewing the contents with the less command, allowing us to scroll up and
down through the file.
If you know the area you are looking at, e.g. storage, then we can launch the tool
with the appropriate switch. Here are the six switch options:
w hardware
r resource
s storage
n network
y system
o advanced options
If we combine the filtering of the output using the above switches along with
a grep filter we can really zoom in on the area we are interested in. An excellent
VMware communities post gives an example of using the storage switch whilst
looking for Pending reservations on LUNs. We are piping the result of the storage
output of esxcfg-infointo the input for grep.
esxcfg-mpath
Manages storage multi-pathing just as the vmkmultipath utility did in previous
versions of ESX Server. In the example below we are using the -l switch to list
the storage and paths.
Disk vmhba1:0:0 (0MB) has 1 paths and policy of Most Recently Used
FC 10:1.0 210000e08b846a72<->5006016930221397 vmhba1:0:0 On active
preferred
esxcfg-resgrp
Used to manage the new ESX feature called resource groups. This command can
add, remove or modify existing resource groups.
esxcfg-hbadevs
The esxcfg-vmhbadevs command is used to list the equivalent Linux device
names for the visible disk devices that the VMkernel references using vmhba
notation.
If we use this command with the –m switch, then we only list the LUNs which
contain VMFS partitions. Alongside the Linux device name, a long unique
hexadecimal value is listed. This is the VMFS volume signature assigned by the
new logical volume manager (LVM).
esxcfg-boot
Used to configure the GRUB options presented at boot time. One thing to note is
that the new esxcfg commands will not run if you boot just into Linux. If you just
want to query the boot settings, you can use the -q switch but this must be
qualified with the keyword boot or vmkmod.
This is also used if you making modifications to VMkernel device drivers defaults.
For example, if you were modifying the queue depth for a fibre HBA, you would
likely be usingesxcfg-module. Then to rebuild the boot image you would enter
After which, you would do a reboot the host to test that the update to the boot
image had worked.
esxcfg-init
Should not be run manually!
esxcfg-nas
The esxcfg-nas command is used to list, mount and dismount NFS exports for
the VMkernel. In the first example we list the NFS datastores which the VMkernel
has mounted.
In the next example, we add a new VMkernel mount to a remote NFS server. This
time we are connecting to the NFS server at IP address 100.100.100.253 and the
name of the exported directory on the NFS server is “/Test”. We are labelling
(from the ESX host perspective) this NFS mount as “NFS02”. This will appear as
the datastore name on the ESX host.
esxcfg-route
If we add an IP address to the VMkernel by adding a VMkernel port, then we can
fully configure that IP stack by also assigning a default gateway. We can view (no
parameters) and set (1st parameter) the VMkernel IP default gateway with
the esxcfg-route command as shown here. In the following example, we view
the current VMkernel gateway (.254) and then change it to a new one (.1)
[root@esx1host etc]# esxcfg-route
VMkernel default gateway is 100.100.100.254
As of ESX 3.5, we have the -a switch which is used to add additional routes to
the VMkernel routing table. We also have the -l switch to list the VMkernel routing
table. In the following example, we list the routing table, then add a new static
route for the 192.168.90.0/24 network and check it has added to the VMkernel
routing table correctly.
VMkernel Routes:
Network Netmask Gateway
100.100.100.0 255.255.255.0 Local Subnet
default 0.0.0.0 100.100.100.1
VMkernel Routes:
Network Netmask Gateway
100.100.100.0 255.255.255.0 Local Subnet
192.168.90.0 255.255.255.0 100.100.100.165
default 0.0.0.0 100.100.100.1
If we want to remove an entry from the VMkernel routing we use the -d switch.
So in the following example, we are removing the newly added route.
esxcfg-vmknic
Used to view and set configure the VMkernel ports on virtual Ethernet switches. A
VMkernel port is a special type of port group on a virtual Ethernet switch which is
used to assign an IP address to the VMkernel. The VMkernel only needs an IP
address for VMotion, software-initiated iSCSI or NFS access.
If you need to create a VMkernel port at the command line, then you need to
create a port group first and then enable it as a VMkernel port. This tool does not
allow you to enable the VMkernel port for VMotion, you must either use vimsh or
the VI client for that.
In the following example, we list the VMkernel ports, then use esxcfg-vmknic to
delete one of them and then list them again.
As of ESX 3.5, we can set the MTU for VMkernel initiated traffic. We should be
aware however that currently Jumbo frames is only technically supported for
VMotion and not iSCSI or NAS, even though it does work. Anyway, if you decide
you want to enable Jumbo Frames for an existing VMkernel port, you are going to
have to delete and recreate that VMkernel port. The MTU size for a VMkernel port
can only be set at creation time. So, continuing our above example, if we wanted
to enable an MTU of 9000 on the port group "NFS Access" we would need to do
the following:
One final note on this utility is about the disable function. If you disable the
VMkernel port, you cannot delete it while in this state. If you want to delete a
VMkernel port, it must be enabled or the call to delete it is ignored.
So far, we have only used this utility to interrogate ESX hosts to determine where
the dump partition has been created. Here is an example of viewing the dump
partition.
# esxcfg-dumppart -l
Remember that the dump partition does not show up when you run
the vdf utility. However it is visible if you run fdisk. In the following example,
we are running fdisk to view the partitions. We can see the dump partition as
c0d0p7, i.e. partition #7. Notice the Id of that partition is "fc", the custom
partition type for VMkernel dump partitions.
# fdisk /dev/cciss/c0d0
esxcfg-linuxnet
There is not normally a command that a virtual infrastructure administrator
should need. The tool is automatically used when you start an ESX server in
troubleshooting mode; i.e. when you start only the service console Linux kernel
and don't start the VMkernel.
When you are working in the service console while the VMkernel is loaded, the
service console's network interface is not called eth0, but is called vswif0 instead.
This is because the service console network interface is provided via a service
console portgroup on a virtual Ethernet switch. If you restart your ESX server
without the VMkernel, then standard Linux drivers and network card management
is used. Therefore the network interface used in troubleshooting mode is called
eth0 - just like any other regular Linux box. This tool is called by starting
troubleshooting mode to replicate the IP parameters assigned to vswif0 to eth0.
esxcfg-linuxnet --setup
--remove
-h --help
esxcfg-nics
This tool can be used to view and configure the speed and duplex settings of the
physical network cards in the ESX Server. This tool can replace the mii-
tool andmodules.conf for network card management.
In the following example, we run the list option to view all physical NICs and their
properties.
esxcfg-swiscsi
ESX server 3 supports both hardware and software initiated iSCSI. For hardware
iSCSI, we can use host bus adapters which perform the TCP offload and so the
VMkernel can just pass SCSI commands to them as normal. The iSCSI hba can
then wrap the SCSI command in IP transport and forward them to the iSCSI
target.
In VI-3, one of the supported iSCSI hardware HBAs is the QLogic 4052. More
information about this particular family of adapters can be found
athttp://support.qlogic.com/support/product_resources.asp?id=964
1. Add a VMkernel port to a vSwitch that has an uplink and route to iSCSI target
2. Ensure service console IP interface has a route to the same iSCSI target
3. Using either the VI Client security profile or the esxcfg-firewall, open a port
in the service console firewall for iSCSI (TCP:3260)
4. In the command line, enable iSCSI with the command esxcfg-swiscsi -e
5. Enable a discovery address with the command vmkiscsi-tool -D -a
10.0.0.99 vmhba32
6. List the targets that were discovered with vmkiscsi-tool -T -l vmhba32
7. Perform a rescan with esxcfg-rescan vmhba32
8. List the iSCSI LUNs with vmkiscsi-tool -L -l vmhba32
If you want to ensure the VI client reflects the changes made at command line, it
is best to restart the vmware management service with the command service
mgmt-vmware restart
The full list of command line options for this command are:
esxcfg-vswif
This tool can manage the Ethernet interfaces of the service console. In a big
change from previous versions of ESX, the Ethernet interface of the service
console is named with the "vswif" prefix and not "eth" prefix as you may be used
to in Linux.
In VI Client we can view the network configuration of our ESX host. Here is an
example of a typical network configuration.
# esxcfg-vswif -l
If we wanted to add a 2nd service console port, we could use this command.
However, all this command will do is turn a regular portgroup into a service
console port and bind an IP address to Linux. So in the following command line
example, we create a portgroup first, and then we turn it into a service console
port with esxcfg-vswif.
So now if we run esxcfg-vswif to list the service console ports, we will be able
to see the original service console port as well as our new one we just created.
We've shown you the graphical representation as well from the VI client so you
can compare.
# esxcfg-vswif -l
A new function was added to esxcfg-vswitch when ESX 3.5 was released at the
end of 2007. This version of ESX server was the first to support Ethernet Jumbo
Frames. This is where the MTU size is increased beyond the default 1500 bytes.
In the following example, we are changing the maximum MTU for vSwitch1.
Configuration Files
/etc/vmware/esx.conf
An all new configuration file for ESX Server 3.x. This file replaces the functionality
of the following configuration files found in earlier versions of ESX.
/etc/vmware/hwconfig
/etc/vmware/devnames.conf
/etc/vmware/vmkmodule.conf
/etc/vmware/netmap.conf
/etc/vmware/vmkconfig
This file should not be copied from one ESX host to another in order to duplicate
configuration, it is unique to the host. The file groups similar settings by using a
notation similar to directories and subdirectories; for example, here is a section
of esx.conf
<output>
/etc/nsswitch.conf
This is the name service switch configuration file. If you need to modify the order
of how names in the service console are resolved, this is the place to make the
change. You can view and edit this conf file as usual. There will be a number of
lines to this file, but the one you are likely to be interested in will start "hosts:"
as shown:
In the above example, the name service will use the /etc/hosts file, then NIS+
and then the DNS name server specified in the /etc/resolv.conf file. If the
application that is trying to use a hostname is using the libc resolver library (by
using the gethostbyname function call) the nsswitch.conf file is used.
However, an application could use its own resolver library. An example of this is
the dig utility for testing DNS lookups - this tool ignores
the /etc/nsswitch.conf file.
/usr/bin/vmware-watchdog
This process watches over the hostd process and restarts it if it crashes.
hostd
This is the daemon that replaces vmware-serverd that was found in the ESX 2.x
products. This is the host management agent and is responsible for a number of
key management functions on an ESX host. If you are having any "host not
responding" type problems, before you even think of an ESX host restart,
consider just a restart of the management agent; it's amazing how often a quick
restart of hostd gets things going again.
/var/log/vmware/hostd.log
The log file for the host management agent.
/etc/vmware/firewall/services.xml
This file contains the definitions for the TCP ports and service names used by the
service console firewall. When we use the esxcfg-firewall command to open
ports based on friendly service names such as sshServer, that name is a
definition in this XML file. A typical service definition in this file looks like
<service id='0000'>
<id>sshServer</id>
<rule>
<direction>inbound</direction>
<protocol>tcp</protocol>
<port type='dst'>22</port>
<flags>-m state --state NEW</flags>
</rule>
</service>
You could modify this XML file to include your own definitions but this is not
recommended by VMware. The VMware management agent (hostd) will load
everything in this file, whether it is valid or not. Also, we have not tested if such a
change would persist through a patching/upgrade, but we suspect not. Duncan
Epping over at Yellow Bricks has done some great testing and documentation in
this space and at the following link demonstrates how to add your own
custom.xml file to the /etc/vmware/firewall directory (using same format
as services.xml) to provide custom port definitions. You can read all about it
at http://www.yellow-bricks.com/2007/12/31/howto-adding-a-firewall-service-
on-esx/. Just make sure you use ids in the file that are different than the ones
in services.xml.
vpxa
This is the name of the VirtualCenter server agent that runs in the service console
of ESX 3.x servers (which was called vmware-ccagent in ESX 2.x). This can be
stopped, started or restarted with the service command
/etc/vmware/vpxa.cfg
This is the XML configuration file for the VirtualCenter Server Agent in the service
console. Here is a typical vpxa.cfg file.
Notice the <loglevel> tag. If you are trying to troubleshoot an issue, then
increasing the logging level is a good idea. We have used the level "verbose",
there could be a higher debug level of logging, but we've not tested that. We
have also found the loglevel trivia, info, warning and error.
/var/log/vmware/vpx/vpxa.log
The log file for VirtualCenter agent in the service console.
vmkfstools
Used to manipulate VMFS and virtual disks at the service console command line.
In ESX2.x we used it most often for import and export operations, where a virtual
disk is converted from monolithic format to sparse format (previously called COW
format). Now we tend to use it in ESX scripted install scripts to automate VMFS
configuration.
VMFS volumes can be spanned across LUNs. We are not big fans of this as it
tends to indicate storage wasn't planned in the first place and now things have
reached crisis! However, they can be useful in certain circumstance
and vmkfstools steps up again.
The -X (case-sensitive) switch is used to extend the size of a virtual disk; e.g. if
you had a 10GB virtual disk and wanted to extend it to 20GB, you could use this
command. The VM would need to be powered off for this to work.
Note that the -X switch specifies the NEW SIZE of the virtual disk and NOT how
much you are extending it by.
If you have used the -X switch before in an older version of ESX server (earlier
than 3.0) it was possible to specify a small disk size; thereby making the virtual
disk smaller. This was dangerous but useful if your partition within the disk did
not consume 100% of the disk size. However, this is not possible
with vmkfstools command found in ESX Server version 3.x.
From ESX 3.5, the size of a virtual disk can now be increased in the VI Client!
VMware are implementing more and more in the user interface, less time needed
in the service console command line...
Previously, the main use of vmkfstools command was to import or export virtual
disks. This would be required if you were deploying templates by hand instead of
using VirtualCenter. It was also the primary method for moving VMs between the
ESX server product and the hosted VMware products such as VMware Workstation
or Server. The reason we say "previously" is that moving VMs between servers or
between VMware products has become much simpler and cleaner by using the
VMware Converter utility. This tool is task oriented and treats the VM as a whole
object, not just the virtual disk files as vmkfstools.
If you do want to import virtual hard disks that are in 2GB sparse format into
monolithic format by hand, then we can use vmkfstools command with the -i
switch.
Notice that the import option requires two parameters, source and destination.
This would not create a VM, but would create the monolithic virtual disk for a VM.
You could then create a custom VM in the VI Client and select the option to "use
an existing disk".
If you want to export a virtual disk you no longer use the -d <type> switch, but
just use the same -i switch and specify the virtual disk type at the destination of
the import. So if you were exporting a virtual disk from VMFS to a ext3 directory
path you could use:
All being well, our storage is well planned out, disks are thick provisioned at
creation and we get no surprises. However, things are not as straightforward as
we always want. The business want changes to the virtual disk sizes, they want
to save money on storage provisioning etc! So, it is possible that a virtual disk
could be fragmented. The vmkfstools command can help us again here. The
undocumented -t switch will show us how many contiguous sections a virtual disk
has. If it only has 1 section, then it's not fragmented.
vmkfstools -t /vmfs/volumes/storage1/vm/vm.vmdk
vmware-cmd
This command has been in ESX for a number of versions and it's functionality has
been extended with each major release. We tend to find that the most frequent
use of this command is to register or power on VMs from the console command
line
If you have a VM that you can't tell if it is powered on or off, you can use the
getstate option
If there is limited space in your VMFS volumes, then you will likely want to know
if any of your VMs are running in snapshot (where the disk writes are going into a
disk delta and not the regular parent virtual disk). It is a nice idea to have a short
script to enumerate the VMs on your host and loop through them to check each of
them to see if they have a snapshot. The vmware-cmd command again helps us
out with this.
vm-support
A great built-in tool which collects all configuration files on an ESX host and builds
a tar archive that can be sent to VMware support so they can have a complete
picture of your system to assist in the troubleshooting effort.
A useful function of this tool is to list running VMs using the -x switch.
<output>
[root@esx1 root]#
Watch out for the creation of empty subdirectories of the name "vm-
support.<pid-of-process>" in the directory where you run this tool with the -x
switch. It is safe to delete these directories. You can't run this command if your
current directory is /proc.
The performance snapshots are archived automatically into a tgz file (a tgz file
just like a WinZIP (R) archive). The tgz archive file name produced is unique to
each time it's run, as the name includes date, time and process id of vm-support.
Before we can actually replay the snapshot performance data in esxtop, we need
to extract the tgz archive. The tar command is used to "unzip" tgz archive files.
[root@esx1 root]# tar -zxvf archive.tgz
To replay the data in esxtop, use the "-R" switch to specify replay mode and
supply the path to the performance capture file produced by vm-support.
esxupdate
This utility is what we use to patch our ESX hosts with updates from VMware. You
can use this tool interactively to install individual patches, or use it to scan your
ESX host to see which patches are required as well as to do a "what-if" install of a
host patch to identify if there will be any problems.
The power of the esxupdate command is realised when you use it with a patch
repository. A patch repository can be exposed to a host via HTTP, FTP or NFS.
If you choose to use the new VirtualCenter Server 2.5 feature called VMware
Update Manager (VUM), then when you perform host scans and remediation, you
are in fact just remotely invoking this utility, it's just you don't see it!
You can use the --explain switch when scanning to provide a greater level of
detail to your host patch scan operation. If for example, the AppFlags for a patch
indicated "c" for conflict, you would probably want to know what exactly the patch
was in conflict with.
/var/log/vmware/esxupdate.log
The log file for the esxupdate host patch utility.
contents.xml
Every ESX patch contains a file called contents.xml. This file describes the
directory structure of the patch bundle contents.
contents.xml.sig
This is a detached PGP signature of the contents.xml file in a ESX patch.
vimsh
This is a superb utility that we use on occasion, particularly when we are creating
scripted builds for ESX. The industry-recognised experts in the functions of this
tool are the folks over at www.xtravirt.com. Where we have found this tool of
unique use is in the enabling of a VMkernel port for VMotion.
However, if you are using ESX version 3.5 then we need to use a slightly different
syntax for specifying the portgroup to enable. We now need to specify using a
vmkxnotation. Trouble is, we don't know which portgroup corresponds to which
vmkx number. So to first identify the mapping of portgroup name to vmk
number, we enter the command
vimsh
Using the vimsh command for enabling VMotion is just 1% of the functionality of
this tool. It's not for the faint hearted and there really is no better source of
information about it than the PDF documents that the xtravirt guys have written.
Find their article here http://knowledge.xtravirt.com/white-
papers/scripting.html .Thanks also to Mike Laverick of RTFM Education
(www.rtfm-ed.co.uk) for documenting the changes in vimsh in version 3.5.
vmware-vim-cmd
This command is a variation on the vimsh command that allows faster execution
as we can invoke this command using the same options we use with vimsh,
however this time we don't end up inside the vimsh shell after execution. If you
use vimsh, after execution you are in a weird shell with a prompt like the
following:
[/] $
Using vmware-vim-cmd is straightforward as you just run the command and you
are returned to the regular bash shell in the service console. For example
vmware-vim-cmd /hostsvc/hostsummary
RPM Utilities
rpm
As ESX service console is based on modified Red Hat Enterprise Linux 3, we can
use the RPM package installation method to add applications to it. However, we
should also point out that it's maybe not the best idea to add software to the
service console. It is best to treat the service console as a dedicated console and
not add applications to it.
If you are unfamiliar with RPMs in Linux, think of them like MSI packages in
Windows.
The rpm command can be used to list and to install RPM-based applications. In
the following example, we are using the command switch (-qa) to list the rpms
installed in the service console.
# rpm -qa
libgcc-3.2.3-53
setup-2.5.27-1
basesystem-8.0-2
tzdata-2005m-1.EL3
glibc-2.3.2-95.37
bzip2-libs-1.0.2-11.EL3.4
etc!.....
If we are only interested in the VMware rpms, then we can just pipe the output
of rpm -qa command into the grep search tool.
VMware-webCenter-esx-2.0.1-32041
VMware-esx-apps-3.0.1-32039
VMware-esx-iscsi-3.0.1-32039
VMware-esx-uwlibs-3.0.1-32039
VMware-esx-vmkernel-3.0.1-32039
VMware-esx-drivers-block-DAC960-2.4.11-32039
VMware-esx-drivers-net-bcm5700-7.3.5-32039
VMware-esx-drivers-net-e100-2.3.40-32039
VMware-esx-drivers-net-pcnet32-1.30c-32039
VMware-esx-drivers-net-tg3-3.43b.1vmw-32039
VMware-esx-drivers-scsi-adp94xx-0.0.5-32039
VMware-esx-drivers-scsi-aic7xxx-6.3.9-32039
VMware-esx-drivers-scsi-lpfcdd-v732-7.3.2.1vmw-32039
VMware-esx-drivers-scsi-megaraid_sas-0.0.2-32039
VMware-esx-drivers-scsi-qla2200-v7.07-7.7.4.1vmw-32039
VMware-esx-drivers-scsi-qla4010-3.24-32039
VMware-esx-drivers-scsi-vmkiscsi-3.4.2-32039
VMware-hostd-esx-3.0.1-32039
VMware-esx-lnxcfg-3.0.1-32039
VMware-esx-perftools-3.0.1-32039
VMware-esx-docs-3.0.1-32039
VMware-esx-tools-3.0.1-32039
VMware-esx-vmkctl-3.0.1-32039
VMware-esx-drivers-block-cciss-2.4.54-32039
VMware-esx-drivers-net-3c90x-1.0.2-32039
VMware-esx-drivers-net-bnx2-1.3.22-32039
VMware-esx-drivers-net-e1000-7.0.33.2vmw-32039
VMware-esx-drivers-net-s2io-1.7.6-32039
VMware-esx-drivers-scsi-aacraid_esx30-1.1.5.1vmw-32039
VMware-esx-drivers-scsi-aic79xx-6.3.9-32039
VMware-esx-drivers-scsi-ips-7.10.17.1vmw-32039
VMware-esx-drivers-scsi-megaraid2-2.10.7-32039
VMware-esx-drivers-scsi-mptscsi_2xx-2.6.34.1vmw-32039
VMware-esx-drivers-scsi-qla2300-v7.07-7.7.4.1vmw-32039
VMware-esx-drivers-scsi-qla4022-3.24-32039
VMware-esx-vmx-3.0.1-32039
VMware-esx-srvrmgmt-3.0.1-32039
VMware-esx-backuptools-3.0.1-32039
VMware-esx-scripts-3.0.1-32039
VMware-esx-3.0.1-32039
VMware-cim-esx-3.0.1-32039
VMware-vpxa-2.0.1-32042
If we then want to know what files are included in the rpm package, we can use
query with the list option to see the files inside. For example, to see the files
/etc/vmware/hostd/config.xml
/etc/vmware/hostd/env/0.xml
/etc/vmware/hostd/env/1.xml
/etc/vmware/hostd/env/vmconfigoption-esx-2.5.0.xml
/etc/vmware/hostd/env/vmconfigoption-esx-3.0.0.xml
/etc/vmware/hostd/environments.xml
/etc/vmware/hostd/esxinfo.vha
.....
rpm -ivfh?XXX
rpm2cpio
If you are wanting to extract a single file from a RPM package but you don't want
to install the RPM, then this is the tool for you. Probably best if you copy the RPM
to a temp directory so when you extract the RPM you can then navigate the
directory structure created in that temp directory to find the file or files you need.
Once you have copied out the file you were after, you can safely delete the
contents of that temp directory. In other words, we have used rpm2cpio to
extract the RPM archive.
Here is an example using the RPM we've used in the previous examples.
i = Restore archive
d = Create landing directories
m = Create previous file modification times
v = verbose
Linux Utilities
/etc/ssh/sshd_config
The configuration of SSH client is stored in the text file /etc/ssh/ssh_config
If you do edit the file to change this setting to Yes, then make sure you restart
the daemon for the changes to take effect using the command:
It is also possible to explicitly allow or deny specific users to the SSH daemon.
The headings in the ssh_config file are DenyUsers and AllowUsers.
su
This command is the switch user utility. Think of it as the command line
equivalent of Windows Fast User Switching!
When it used without parameters, we are specifying to switch to the user root.
However, we can use the su command to switch shell to any user account that we
know the password of. In the first example, we are logged in as the
user kevin and we are switching to user ali.
[sara@esx1host sara]$ su -
Password:
[root@esx1host root]#
If we restrict the built-in user account root from logging in over the SSH
protocol, then we are forcing remote users to authenticate as themselves and
then su to run privileged commands if need be, thus leaving a decent audit trail.
The downside being that those users would still know the root account password.
If you would like to restrict the use of the su command, then we can limit it to the
members of a specific group called wheel. This group is defined in
the /etc/group file by default and it's membership can be modified by root. In
order to limit su to the wheel group members we need to modify a configuration
file called /etc/pam.d/su
There is a single line in this file that needs to be uncommented to limit the use
of su. The line is shown below as it appears it that file, all that is required is the
removal of the # symbol at the start of the line.
sudo
The downside of the su command is that the operators who elevate their privilege
to root are now root. They have full privilege, they know the root password, there
is no granularity of delegation of privilege.
The vmkfstools command would then run under the security context of
the root user. The superb feature of this tool is that the user ali does not need
to know or supply the root password to be able to run the delegated command.
Further, we keep an audit trail of when sudo was invoked in /var/log/secure.
The sudo tool uses the lookup file /etc/sudoers to determine which users can
perform which commands. We do not edit this file with a regular text editor like vi
or nano, instead we use the tool visudo.
visudo
This is the vi text editor with extras. When launched, it automatically opens and
locks for exclusive edit, the /etc/sudoers file. The point of visudo is to ensure
we always edit the right file as the location of the sudoers file differs between nix
distributions, but this command is constant and will utilise the right sudoers file
for the distribution being used.
A great benefit of using visudo over regular vi, is that it performs some basic
syntax checking for us!
/etc/sudoers
The text file that contains the sudo users and the rules that apply to them. The
first "ALL" relates to all machines (useful if this is a network wide file). Otherwise,
this could be the hostname of the one machine we are trying to run the command
on. In the following example we are allowing the user "alistair" to run the kill
command, all of the commands in the directory /usr/bin and any commands in
the directory /usr/sbin/alistair
In the following line added to the /etc/sudoers file, we are allowing the
user sara to run the esxcfg-firewall and esxcfg-vswitch command.
You can use aliases within this file to group together users, hosts and commands.
Although, rather than maintaining a static configuration file on each ESX host, it
would be better to customize the sudoers file during host deployment and include
Linux groups. For example, if we wanted to delegate a set of commands to those
Linux users who belong to a Linux group, for example, wheel, then we can use
the % operator to leverage those group definitions, thus avoiding static user
aliases.
The best source we've found so far on detailed use and background of sudo can
be found at http://aplawrence.com/Basics/sudo.html
w
Great for viewing logged on console users.
[root@esx1host firewall]# w
12:07:45 up 4 days, 2:16, 3 users, load average: 0.00, 0.00,
0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 - Fri10am 4days 0.02s 0.02s
-bash
root tty2 - Fri 9am 4days 0.06s 0.06s
-bash
root pts/0 remote7.lab.vmwa 9:29am 0.00s 0.06s 0.00s w
who
This command allows use to view who is logged onto the service console either
interactively at the console or via an SSH session. The who command without
parameters gives us the basics.
[root@esx1host firewall]# who
root tty1 Jul 25 10:30
root tty2 Jul 25 09:56
root pts/0 Jul 29 09:29 (remote7.lab.b2v.net)
If we want to see all the details of users we can use the -a switch to show all
data. We tend to combine -a with -H (i.e. -aH) to display column headers making
it easier to read and interpret.
vmkload_mod
This command will load and unload VMkernel modules on the fly. The results of
this load/unload will happen as you type it and will only be valid for the current
booted session. So this command is superb for troubleshooting as we can load
and unload modules, e.g. network drivers.
In the following example, we are examining the options for the Intel network
driver (e1000) with the -s (show parameters) switch and then unloading it using
-u (thereby interrupting network operations on that physical interface
temporarily) and then loading it again with a new option. Notice to load a
VMkernel parameter, we just supply the module name to the vmkload_mod
command as a parameter listing any module-specific options as further
parameters.
If you were experimenting with a driver setting to determine the right setting for
the environment, then you may be loading and unloading a module a number of
times. Once we were happy we had found the correct setting, it is likely we would
want to make this change persistent, i.e. we would want it to take effect for each
time the kernel module is loaded at server boot time. For that, we should
use esxcfg-module command.
minicom
This is a great utility for talking to serial attached devices; we think of it as
HyperTerminal for Linux. Where we have found this particularly useful is for
command line administration of your storage array. For example, if you had an
HP MSA1000 attached to you ESX host and attached the serial cable to the unit
and your host, then you could manage LUN presentation from the service console
command line.
Minicom uses a configuration file to determine bit rates etc. This configuration file
is placed in the /etc directory. We normally create the file with a meaningful
name e.g.minirc.com1, so to launch the tool we enter
# ./minicom com1
pr port /dev/ttyS0
pu baudrate 19200
pu bits 8
pu parity N
pu stopbits 1
pu rtscts No
vi
We can't talk about the command line without talking about vi. This is the simple
but powerful text editor in Linux and UNIX. People tend to love it or hate it. Either
way, it's nearly always there in any *nix implementation and just by memorising
a few commands you can be up and running with it. If you can use Windows
Notepad, you can use vi!
vi filename
The first thing that throws you is that to enter text into your file, you need to
press "i" for Insert mode. You can then enter your text just as any other text
editor. When you are done with text entering, just press the Escape (Esc) key to
come out of insert mode. If you are happy with your file, then we need to Write &
Quit (wq). To enter commands in this command line editor, rather than having
menus, we have a command prompt in the application. To reach the vi command
prompt, simply enter ":" - the colon character which will automatically place your
cursor at the bottom of the session. Here you can enter the "wq" command to
write and quit the editor. That's it!
SHIFT ZZ Quit the editor and save any changes made - just a fast way of
doing ":wq"
Esc key Exits the current mode, e.g. out of insert mode back to view mode.
These commands are just extra if you have the inclination to learn!
/ search - if you entered /failed then the cursor would move to the
first instance of "failed in the text
$ jumps to the end of the opened file
yy copy - it's y for yank!
dd delete a line (cut) if you precede this with a number e.g. 8dd,
then it would delete 8 lines
p paste
%s/old/new/g substitute any occurrences of the world "old" with the world
"new"
There are some great web sites which document the features of vi in superb
depth, one of them is the staff site at University of Washington which helped me.
Their site is athttp://staff.washington.edu/rells/R110/
nano
Another text editor, more friendly than vi but you should use –w to avoid word
wrap.
/etc/ntp.conf
[root@esx7 firewall]# cat /etc/ntp.conf
# Prohibit general access to this service.
restrict default ignore
#
# Drift file. Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the
file
# by creating a temporary in the same directory and then rename()'ing
# it to the file.
#
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
#
# Authentication delay. If you use, or plan to use someday, the
# authentication facility you should make the programs in the
auth_stuff
# directory and figure out what this number should be on your
machine.
#
authenticate yes
#
# Keys file. If you want to diddle your server at run time, make a
# keys file (mode 600 for sure) and define the key number to be
# used for making requests.
#
# PLEASE DO NOT USE THE DEFAULT VALUES HERE. Pick your own, or remote
# systems might be able to reset your clock at will. Note also that
# ntpd is started with a -A flag, disabling authentication, that
# will have to be removed as well.
#
keys /etc/ntp/keys
/etc/ntp/step-tickers
If you have a single time source configured for your service console, then this file
will have just 1 line, similar to the following:
server 192.168.1.100
ntpdate
If you want to synchronise your service console clock with the defined time
server, you can use this command with the -u switch.
ntpdate -u timeserver.local
ntpq
This queries the state of the ntp service. Watch for the back ticks used in the
parameters, they are not single quotes!
date
If we are checking the time and date of our ESX Service Console, then the date
command is very useful. Just entering the "date" command returns what the
service console thinks the current date is.
If the date is incorrect and you wish to reset it you would enter the command
with the -s switch and specify date in mm/dd/yyyy format.
Once you have set the date, you will want to ensure that the hardware clock
matches your newly entered date. We can do this with the hwclock command
described below.
hwclock
We can use this command to synchronise the server hardware clock with the date
we set in the service console. If you enter the command with no parameters then
the value of the hardware clock is displayed.
# hwclock
If we want to synchronise the hardware clock with the service console date and
time, we use the following:
# hwclock --systohc
cal
Display calendar for current month or set of months. The following command
displays 3 months, current month and the month before and after.
# cal -3
March 2006 April 2006 May 2006
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 1 1 2 3 4 5 6
5 6 7 8 9 10 11 2 3 4 5 6 7 8 7 8 9 10 11 12 13
12 13 14 15 16 17 18 9 10 11 12 13 14 15 14 15 16 17 18 19 20
19 20 21 22 23 24 25 16 17 18 19 20 21 22 21 22 23 24 25 26 27
26 27 28 29 30 31 23 24 25 26 27 28 29 28 29 30 31
30
Surprisingly useful!
passwd
Used to change the password of the currently logged on user (use the command
with no parameters) or for changing the password of a named user account
(supply the user name as a parameter).
passwd <user>
Remember that passwords are not stored in the /etc/passwd file (that's where
users are defined) but are actucally stored in the file /etc/shadow
If you are ever needing to reset an unknown root account password, then it is
this utility you would run after booting into Linux single user mode.
VMware HA
AAM
AAM is the Automated Availability Manager that runs in the service console when
you create a VMware High Availability (VMware HA) cluster. The VMware HA
feature was previously known as DAS (Distributed Availability Services) but we
don't mention that anymore.
This is a piece of licensed Legato software which itself has been renamed to EMC
AutoStart.
This component has a very high dependency upon fully functional host name
resolution. So before you enable VMware HA, check the following files
/etc/hosts
/etc/FT_HOSTS
/etc/resolv.conf
/etc/vmware/esx.conf
to ensure accuracy. One thing you can do to check the name resolution
functionality before enabling HA is run
hostname -s
to return the short name of the service console. If this fails, then the HA
configuration WILL fail.
The log file for VMware HA in ESX 3.0.x can be found in the service console in the
directory
/opt/LGTOaam512/
/opt/VMware/
To avoid split brain scenarios, an ESX server can determine if it has become
isolated from other servers and we can configure that servers' isolation response.
If the AAM component loses contact with the other nodes in the HA cluster, it
attempts to contact the configured default gateway for service console using ICMP
echo request (PING). If this fails, then the ESX host is isolated. If your default
gateway suppresses ICMP echo requests, then we can configure an alternate IP
address called thedas.isolationaddress. From ESX 3.5, you can configure
multiple isolation addresses so that you can configure a host with more that one
address to attempt contact with before declaring itself isolated.
/opt/LGTOaam512/bin/ftcli
This utility allows you to view the active nodes in an HA cluster and the managed
IP addresses. This utility will help you determine whether the HA agent is in a
running state and which IP addresses are visible between those managed hosts.
/etc/FT_HOSTS
This file is created when HA is enabled and is a copy of /etc/hosts. If you have
problems with name resolution and configuring HA, you can safely delete this file
and reconfigure that cluster node for HA again. FT_HOSTS will be re-created.
Networking
hostname
This utility displays the service console hostname. There are some useful switches
to this command
and
ifconfig
Used to determine what IP address you have, the equivalent of
the ipconfig command in Windows. You can use the command without
parameters to view all interfaces, or you can be interface specific, e.g.
Notice this is a quick way of viewing your COS virtual MAC address alongside the
IP address. Also note, you don't see the additional optional IP parameters like
gateway and DNS servers.
ping
Our favourite IP connectivity tool; I love the name Packet InterNetwork Groper!
This tool uses ICMP to send an "echo request" and looks for an ICMP e"cho
reply" . There are a couple of very useful switches we can use with ping, the most
common one we use is -c to specify count. The Linux ping command keeps
pinging continually by default (Windows needs -t to do that). So if we only want 4
pings, we specify -c4.
[root@esx1host root]# ping 192.168.93.200 -c 4
PING 192.168.93.200 (192.168.93.200) 56(84) bytes of data.
64 bytes from 192.168.93.200: icmp_seq=0 ttl=63 time=0.507 ms
64 bytes from 192.168.93.200: icmp_seq=1 ttl=63 time=0.458 ms
64 bytes from 192.168.93.200: icmp_seq=2 ttl=63 time=0.448 ms
64 bytes from 192.168.93.200: icmp_seq=3 ttl=63 time=0.538 ms
Remember that often firewalls block ICMP echo requests, so not getting a reply
doesn't mean your host is down!
This tool relies on correct ARP functionality. So again, what looks like
a ping failure, may in fact be a local ARP issue and unrelated to the destination
address of the ping operation performed. If you are testing MTU sizes, you can
force ping not to fragment with the -f switch.
XXX
We can set the ttl with this - poor mans traceroute!
vmkping
This ping makes use of IP stack of the VMkernel rather than the Linux network
stack in the service console. So if you are trying to troubleshoot VMotion, iSCSI or
NAS issues where the VMkernel is directly using its own IP (a VMkernel port). We
supply the IP address of the destination as a parameter, just as we do with
regular ping.
[root@esx1host root]# vmkping 192.168.93.200
64 bytes from 192.168.93.200: icmp_seq=0 ttl=63 time=0.871 ms
64 bytes from 192.168.93.200: icmp_seq=1 ttl=63 time=5.079 ms
64 bytes from 192.168.93.200: icmp_seq=2 ttl=63 time=22.754 ms
Be aware this tool makes use of the service console DNS, so if there are problems
there, try vmkping using the IP address of the destination rather than hostname
to ensure that any errors you see are unrelated to name resolution problems in
the service console.
If you use -D it will ping all important stations (own interface, iSCSI and default
gateway).
/sbin/arping
This is a similar utility to ping, but uses Address Resolution Protocol (ARP) and so
the result will only be for local subnet resources, either another host or a
gateway.
Notice in the reply we see the MAC address of the target for the
[root@esx1host]# arp -a
vc.lab.taupoconsulting.com (192.168.1.200) at 00:0C:29:8D:F3:65
[ether] on vswif0
remote7.lab.taupoconsulting.com (192.168.1.70) at 00:50:56:84:19:56
[ether] on vswif0
remote7.lab.taupoconsulting.com (192.168.1.70) at 00:50:56:84:19:56
[ether] on vswif0
It's unlikely you will need static arp entries, but it can be done using the -s
switch.
ethtool
This command can be used to view and configure the Ethernet interfaces in your
ESX host. We didn't use this tool very often until ESX 3.5, when we started to
work with Distributed Power Management (DPM); an experimental feature of DRS
clusters.
The output of this tool provides a load of information about the network cards,
but of particular interest now is the support for Wake-on-LAN (WoL). DPM makes
use of this NIC feature and so we need to check that our NICs both support the
function AND have the function enabled. The ethtool allows us to view and set
this functionality.
# ethtool vmnic1
Settings for vmnic1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes
If we noted that our NIC supported WoL but it was not currently enabled, then we
could use this tool to effect the change.
tcpdump
Neat tool for doing network captures at the service console command line. The -i
switch is used to specify the interface to be used for the capture. This is
important in ESX server, as the default interface for this tool is to use
interface eth0, which doesn't exist for us; we have vswif0 as our default
Ethernet interface.
The format of the FILE can be read with another popular tool; WireShark!
If you need to write a filter for the capture, there are reserved characters that
need escape characters before them, e.g. parenthesis.
ip
This is a very powerful command and we don't often need it unless we are
network troubleshooting at the command line.
We can see neighbours on an interface, which is really another view of the arp
cache.
route
This shows and allows editing of the routing table in the service console. If we use
the route command with no parameters, the Linux routing table is displayed. If
this is taking a long time, this could be due to DNS look ups, so you can use the -
n switch to force numeric (no name resolution).
[root@esx1host root]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref
Use Iface
192.168.90.0 0.0.0.0 255.255.255.0 U 0 0
0 vswif0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0
0 vswif0
0.0.0.0 192.168.90.254 0.0.0.0 UG 0 0
0 vswif0
Note, the add option to the route command was not added in ESX until version
3.5.
tracepath
As the traceroute command is not present in the ESX service console by default,
we should be aware of some alternative tools. This tool traces the path to the
specified destination (supplied as a parameter) discovering the Maximum
Transmission Unit (MTU) along the path. It uses a random UDP port by default,
but can be modified to use a specified port (2nd parameter).
[root@esx1host root]# tracepath 192.168.170.201
1: esx1host (192.168.90.7)
0.173ms reached
Resume: pmtu 65535 hops 1 back 1
netstat
The output of netstat produces more information than just network sockets, so
we need to narrow the query to just tcp and udp protocols.
if you use -n to not resolve hostnames and protocol ports to service names
The -p switch is extremely useful for determining which processes are using those
sockets.
nslookup
The nslookup tool is most often used to check forward name resolution. It can be
used interactively or in a dedicated shell. If used interactively, then simply supply
the name of the host your are looking up as a parameter.
The nslookup tool can also be used for reverse resolution by supplying IP address
as a parameter
dig
This tool is a replacement for nslookup in Unix and Linux environments. It is
fantastic for displaying exactly what is happening when you are doing a name
lookup to DNS. We can see our query, the answer, the authority all in one output.
The dig tool can be used for reverse lookup with -x switch. This tool does not use
standard libc name service lookup and therefore does not refer
to /etc/nsswitch.conf. It goes directly to the DNS servers listed
in /etc/resolv.conf. Note, if you have multiple nameserver entries
in /etc/resolv.conf we only query the 2nd or 3rd entry if the 1st or 2nd cannot
be contacted. If the 1st nameserver responds with an unknown host reply to the
query, we stop and don't query the remaining nameservers.
rpcinfo
Can be used to verify services at a server. We find this useful for verifying if a
server is running NFS v3 over TCP.
showmount
This command is used by a NFS client to see what directories are being exported
by a NFS server.
This command can be specified with the hostname name or IP address of the NFS
server holding the exported directories. Remember that by default the service
console will block nfsClient traffic. You will need to use esxcfg-firewall to
open up that port. Also remember when you are accessing NFS servers from the
service console you are going out via the Linux network stack; this is not the
same operation as adding an NFS datastore, where the VMkernel connects to NFS
via its own stack on its VMkernel port.
portmap
If you are wanting to mount a NFS export on a remote system from the service
console, you do not need all the nfs server daemons running. All you need is the
portmap service. You can start it with
And you can ensure this service is launched at boot time using
the chkconfig command. Also remember that by default nfsClient is blocked by
the service console firewall.
mount
It really helps to be able to do simple mounts of remote systems using NFS and
the mount command. We can tell mount what type of file access protocol to use
with the -t switch, e.g. -t nfs or -t smbfs, however if you are working just with
nfs, you can safely omit this
vcbSnapshot
vcbMounter
If you want to perform image backups of running virtual machines from the
service console command line, then this is the command for you. In a lot of ways
this is the replacement of vmsnap.pl found in previous versions of ESX.
vcbMounter.exe
This command is the core component of VCB which runs on the VCB Proxy server.
vcbExport
mountvm.exe
This utility is only found on the VCB proxy server.
vcbCleanup
This command is new to 3.5/2.5 and allows a backup operator to cleanup the VM
snapshot state should a VCB backup fail before completion.
backuptools.conf
If you don't want to specify user credentials on the command line, you can store
the credentials in this file in the service console.
vcbRestore
When virtual infrastructure administrators first start using VCB, questions arise
about how they can restore their VCB archives. The vcbMounter.exe command on
the backup proxy performs the backup of the live VM, but where is the utility to
perform the restore operation? The answer is, you cannot restore directly from
the VCB Proxy server into VMFS, the proxy server has read-only access to the
VMFS datastores. The vcbRestore command exists in the ESX host service
console. Therefore we need to have the VCB VM archive accessible to the
vcbRestore command to perform a restore. The archive could be copied to the
service console with a tool such as WinSCP or Veeam FastSCP, but often,
administrators will simply perform an NFS/CIFS mount from the service console
to the location where the VCB archive is.
vcbRestore /remotenfs/backups/vm
vcbUtil
This command is only in the service console, not on the backup proxy server.
/proc
The volatile /proc directory hierarchy that can be treated as a file system but is
actually held in RAM. We can interrogate the files and directories in /proc to find
out some great information about the running of the service console.
/proc/vmware
The volatile /proc/vmware directory hierarchy that can be treated as a file system
but is held in RAM. We can interrogate the files and directories
in /proc/vmware to find out some great information about the running of the
VMkernel.
/proc/vmware/migration/history
This is an awesome file to reference when troubleshooting VMotion issues.
/proc/scsi
If you want to check which SCSI devices are visible, cat this file.
/proc/vmware/sched/ncpus
This is an in-memory file displaying the number of processors (ncpus) in the ESX
server. This is a very useful file to inspect when you are unsure how many
physical processors you have and if hyper-threading is enabled.
# cat /proc/vmware/sched/ncpus
4 logical
2 cores
2 packages
You can also get CPU package information from the top three lines of esxtop.
/proc/scsi/qla2x00
If you are using QLogic fibre host bus adapters, this
vpxd.exe
This is the process name of the Windows service that is the core service running
on the VirtualCenter management server.
If there are problems with the VirtualCenter service starting and then stopping
almost immediately or a few seconds later, then check your ODBC database
string and then the health of the the database server. We have seen this when
the database runs out of disk space; check if the log space is full on the DB
server, many clients forget about regular backup of this database. When
troubleshooting the VirtualCenter service you can try VirtualCenter in stand-alone
mode. This is done by invoking the following command at the Windows command
line
vpxd -s
You will get interactive logging of the start-up activity helping you to pinpoint
where the problem is.
If all else fails, you can always re-initialize the VirtualCenter database, however
we would not recommend this. By re-initializing the VirtualCenter database you
are wiping out all VC data!! If you do want this, then use the -b command switch
to vpxd.
vpxd.cfg
This is the VirtualCenter management server configuration file which the VC
service reads at start-up. (Ok, so we are extending this command line guide to
cover the VirtualCenter server now as well as the ESX host!)
<vcp2v>
config.vcp2v.dontStartConsolidation = true
</vcp2v>
You can get rid of the Guided Consolidation feature after VirtualCenter install by
executing the following at the Windows command line on the VirtualCenter
server.
Thanks again to the folks over at Yellow Bricks for that one!
vpxd-#.log
There will be up to log files for VirtualCenter server. The log data is rotated across
10 log files numbered 0-9. Once a log file reaches 5MB, the next one is used.
vpxd-index
This file is the index file which indicates which of the numbered vpxd-#.log files is
the current active one. This file is found on the VirtualCenter management server.
VirtualCenter logs are rotated across 10 log files numbered 0-9.
C:\Program Files\VMware\Infrastructure\VirtualCenter
Server\tomcat\conf\tomcat-users.xml
There should be no reason for you to look at this file on the VirtualCenter server,
however if you want to double check the Apache Tomcat configuration and status
using the Tomcat Manager webpage, then you'll need to.
and then restart the Apache Tomcat service. You can now login to Tomcat
Manager.
vum-proxyAuthCfg.exe
The Update Manager component of Virtual Infrastructure is new to version 2.5.
This component allows the patch management of Windows & Linux guests as well
as ESX hosts. When installing the Update Manager component, the Windows
installer package prompts the operator if they wish to use a proxy server to
connect to the Internet, the only options are proxy IP address and port. If your
proxy server requires authentication, then this tool must be run to supply the
proxy server credentials.
vci-integrity.xml
This is the primary configuration file for the VMware Update Manager (VUM). This
file is read at start-up of the UM service and if the XML file is manually edited,
then the service should be restarted for the change to take effect.
One of the main reasons you may want to edit this file is if you wish to change
the directory that patches are downloaded into, i.e. the patch store. The default
location is on the C:\ drive which may not be adequately sized; there are a
considerable number of patches to download to offer OS and application patch
remediation, totally many gigabytes.
vmware-umds.exe
This is the VMware Update Manager Download Service. If you don't want the
server where Update Manager is installed on to actually connect to the Internet
and do the patch downloading, then UMDS is for you. Maybe you don't want the
load of update downloads on the UM server or maybe the UM server is on a
subnet that can't reach the Internet. Anyway, the UMDS installs on a Windows
server (that is not the same server as UM) and doesn't create a start menu
program group.
vmware-umds --download
Once the updates are downloaded, we can export them. This means we copy the
patches from the download directory to another path. The intended purpose of
exporting is to copy all or a subset of the downloaded patches to a location that
will then be made available to the Update Manager server.
vmware-umds -E e:\exportedupdates
At this time UMDS does not support NFS/CIFS shares for the export operation.
This is related to a permissions issue and the SYSTEM account. The export
function is intended to be used to copy the downloaded data to CD/DVD or USB
stick. That's not to say you can't export to a CIFS share, it's just that's currently
unsupported.
vmware-updateDownloadCli.exe
This tool is run on the Update Manager server to import the patches made
available from the UMDS export. So if you had a DVD burned which had all the
updates that was inserted to the UM server and available as drive Z:
Remote Command Line Interface (RCLI)
These three options bring the ability to run a subset of the commands available at
the service console remotely without having to grant ssh access to the actual
console. This RCLI interface provides the ability for users of VMware ESX Server
3i (hardware embedded hypervisor) to run the esxcfg commands.
Further, this interface is the only interface that a storage VMotion can be invoked
from. To download the RCLI appliance, Windows installer or Linux installer,
visithttp://www.vmware.com/download/download.do?downloadGroup=VI-RCLI an
d accept the EULA to reach the download links page.
Update July 2008 - the RCLI has been updated for ESX 3.5 Update 2. It is well
worth getting the new RCLI tools as VMware have updated them significantly to
be more complete as well as adding support for their use with regular ESX 3.5!
svmotion
This command is run from an RCLI interface to perform a live migrate of a VMs
storage from one datastore to another, known as storage VMotion. In a storage
VMotion, only the virtual disks of the VM move, unlike a regular VMotion, the VM
remains running on the same host.
svmotion --interactive
You can also move disks independently of the virtual machine. If you
want the disks to stay with the virtual machine, then skip this
step..
Would you like to individually place the disks (yes/no)? no
Disconnecting.
So, from the above example, you can see that a storage VMotion run interactively
is quite straightforward. When mistakes can creep in is when you are prompted
for source and destination datastore names. The source datastore name requires
square brackets [] around the name, followed by a space character and path to
vmx file, whereas the destination prompt only requires the datastore name, this
time without square brackets!
If you want to script this command, then the inputs can be supplied as
parameters to the svmotion command.
If you don't want to include user data in the command, then you can combine this
method with an environment variables file called ./visdkrc
/etc/.visdkrc
This is a hidden file that you can create in the RCLI appliance which stores the
parameters you wish to use when running commands against a host.
VI_SERVER = vcserver.taupoconsulting.com
VI_USERNAME = Administrator
VI_PASSWORD = vmware
VI_PROTOCOL = https
VI_PORTNUMBER = 443
ESXi 3.5
DCUI
Ok, the DCUI isn't a command, but you will hear people talking about it, so we
thought it was worth an entry! This is the Direct Console User Interface, in other
words, when you go to the physical console of an ESXi host, what do you interact
with. The DCUI looks and feels like a BIOS management system with menus
navigated using the keyboard.
unsupported
To reach a command line interface on an ESXi host, you could enter the
command unsupported while viewing virtual terminal 1 (tty1); note that there
will be no local echo while you type this command.
Be aware that there is no promise that this option will be present in subsequent
versions of ESXi and that what you do whilst in this version is exactly as
described; i.e. unsupported!
dropbear
This is a small SSH2 server and client available in the unsupported command line
of ESXi. You need to manually enable it by editing /etc/inetd.conf and
removing the comment character (#) from the start of the line relating to ssh.
This is covered really well in a video over at Richard Garsthagen's
site http://www.run-virtual.com/?p=223 as well as on Dave Mishchenko's
site http://www.vm-help.com/
busybox
When you get into the ESXi shell you will find you have a number of common
Linux utilities available to you. You might be forgiven for thinking you have a
service console! Actually the ESXi command line is highly limited shell which
doesn't let you do much. However, busybox gives you a set of regular tools like vi
in a single process. There is more information about busybox at www.busybox.net
vsish
The VSI shell is the ESX 3i equivalent of interrogating the service
console /proc nodes. You can view the state of running VMkernel worlds and
drivers.
Virtual Hardware
Build Numbers
ESXi
ESXi 4.0
ESXi 3.5 Installable Update 3 Build 123629 - 6th November 2008
ESXi 3.5 Installable Update 2 Build 110271 - 13th August 2008
ESXi 3.5 Installable Update 1 Build 82664 - 10th April 2008
ESXi 3.5 Installable Build 70348 - 10th January 2008
ESX4
ESX3
Consolidated Backup
Converter
Converter 3.0.3
Converter 3.0.2 Update 1 (Standalone Enterprise Edition) Build 62456 - 3rd
December 2007
Converter 3.0.2 (Standalone Enterprise Edition) Build 59994 - 18th October 2007
Converter 3.0.1 (Standalone Enterprise Edition) Build 44840 - 26th April 2007
Converter 3.0.0 (Standalone Enterprise Edition) Build 36853 - 29th January 2007
VMmark