SER1289BE

vSphere Troubleshooting
t i o n
Tips and Tricks i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

Javier Menendez & Francis Daly

#VMworld #SER1289BE
Disclaimer
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product. t i o n
tr ibu
r dis purchase orders, or
• Features are subject to change, and must not be included in contracts,
o
sales agreements of any kind.
a t i on
c u b li
• Technical feasibility and market demand will affect o p
r final delivery.
o t f
• n
Pricing and packaging for any new technologiest : N or features discussed or presented have not
ont e
been determined.
17 C
2 0
o r ld
VMw

#SER1289BE CONFIDENTIAL 2
Agenda

Use of the VMware vSphere Command-line Interface (vCLI),

1 t i o n
ibu
the ESXi Shell, and Log Files to Diagnose and Correct Problems
i s tr
or d
2 t ion
Troubleshooting Networking Issues
bli c a
r p u
fo
3
nNot
Troubleshooting Storage Issues
t :
ont e
4 1 7 CvSphere HA Cluster and Performance Issues
d0
Troubleshooting
2
or l
VMw

#SER1289BE CONFIDENTIAL 3
Scope

t i o n
i s tr ibu
or d
t io n
bli c a
r p u
o t fo
nt: N
o n te
1 7 C
2 0
w o rld
V M

This Presentation covers portions of the Troubleshooting Workshop:

For more information, go to http://vmware.com/education
Certain troubleshooting topics are addressed in the VMware vSphere: Fast Track [6.5] course. 4
 t i o n
i s tr ibu
or d
Troubleshooting Overview t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Troubleshooting Process
The troubleshooting process consists of the following tasks:

t i o n
i s tr ibu
or d
t ion
Defining Identifying the cause b li c a Solving
p u
the problem of the problem
t f or the problem
: N o
t e n t
• Identifying symptoms 17 Co• nIdentifying possible • Identifying possible
d 2 0
• Gathering
w o rl causes solutions
informationVM • Determining the root • Implementing the
cause best solution

#SER1289BE CONFIDENTIAL 6
Solving the problem …
Can I do it with the Web Client or should I use the esxi shell?

t i o n
tr ibu
What are my choices:
or d i s
t ion
bli c a
r p u
• Can I fixNtheo fo
t problem with the Web, Host or
e n t :
ont
vSphere
C
Client?
1 7
r ld 20 • Do I want or need to use the command
Mw o line utilities?
V

#SER1289BE CONFIDENTIAL 7
 t i o n
i s tr ibu
or d
a t i on
Troubleshooting Tools and pChoices u blic
t f or
: N o
t e n t
C on
2 0 17
o r ld
VMw
Command-line Troubleshooting Choices
Choices to obtain command-line access on a VMware® ESXi™ host:

t i o n
i s t r ibu
VMware vSphere® ESXi™ Shell vSphere or d
t i n
oAssistant
Management
ic a bl (VMA)
p u
t f or
:N o
• Can be access via:
t e n t • Appliance deployed via ovf
C on
1 7
r ld 20
• 1. The DCUI (if enabled via ALT + F1) • Contains vCLI package
o
VMw
• 2. Via putty after enabling ssh. • Does NOT use port 22.

#SER1289BE CONFIDENTIAL 9
The esxcli Command

The esxcli command

offers the following t i o n
namespaces, i s tr ibu
or d
as well as many new
t ion
esxcli functions added in bli c a
r p u
the vSphere 6.5.
o t fo
nt: N
o n te
7 C
If working with vSAN,
r ld 201
check out the new Mw o
esxcli
V
vsan debug namespace

#SER1289BE CONFIDENTIAL 10
Example: The esxcli vsan Namespace

t i o n
i s tr ibu
or d
The vSAN related
t ion
commands are found in bli c a
r p u
esxcli vsan
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER1289BE CONFIDENTIAL 11
New commands added with every new vSphere release.

Ex: The esxcli vsan debug Namespace

t i o n
i s tr ibu
or d
The new vSAN debug
t ion
related commands are bli c a
r p u
amazing to discover the
o t fo
state of your objects, vms, nt: N
etc. o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 12
Tip: Remember the esxcli esxcli command list command

esxcli esxcli command list t i o n

displays all the command
i s tr ibu
options. or d
t ion
bli c a
r p u
o t fo
To quickly find the command
nt: N
needed, combine it with the
o n te
grep command
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 13
The VMA’s vicfg-* Commands

vicfg- [tab] [tab]

• Commands with the vicfg- prefix enable you to manage t i o n
i s tr ibu
your storage, network, and host configuration.
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M
• Type vicfg- followed by the TAB key twice so see all the available commands.
• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.

#SER1289BE CONFIDENTIAL 14
The esxi shell’s esxcfg- Commands are still there

esxcfg- [tab] [tab]

• Commands with the esxcfg- t i o n
i s tr ibu
prefix enable you to manage
or d
your storage, network, and
t ion
host configuration. bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M
• Type esxcfg- followed by the TAB key twice so see all the available commands.
• Remember the –h (help) option to master these commands.

#SER1289BE CONFIDENTIAL 15
Tip: Remember the “-h” option for esxcfg-* commands

Use the –h option (for help)

t i o n
to see options for this
i s tr ibu
particular command and in or d
t ion
some cases, even
bli c a
examples on how to use the
r p u
command. o t fo
nt: N
How many times did you o n te
17 C
struggle to create a
2 0
ld
orcli?
vmkernel port with the
VMw

#SER1289BE CONFIDENTIAL 16
Manipulating virtual machines with the cli from the V.M.A.?

vmware-cmd
t i o n
• The vmware-cmd command is used to interact with virtual machines.
• vmware-cmd
i s tr ibu
– The vmware-cmd command is exclusively used for virtual machines. o r d
t io n
bli c a
p u
t f or
: N o
t e n t
C on
2 0 17
o r ld
VMw

#SER1289BE CONFIDENTIAL 17
Example: Listing your Virtual Machines

vmware-cmd -l
t i o n
• Lists the virtual machines that are located on the target host.
i s tr ibu
Lists virtual machines by path to the .vmx file.
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 18
Tip: Learn More On Your Own

Type vmware-cmd --help to vmware-cmd allows you to The esxi hostnhas a similar
u t i omanipulate vms
see what the command allows start a vm, take snapshots, command
i s t r ib to
you to do. register a vm, etc.
o d vim-cmd vmsvc.
r called
a t i on
u bli c
or p
o t f
e n t:N
on t
1 7 C
r ld 20
Mw o
V

#SER1289BE CONFIDENTIAL 19
Manipulating virtual machines from the esxi shell

t i o n
i s tr ibu
or d
vim-cmd vmsvc can t ion
perform similar bli c a
r p u
tasks from the o t fo
esxi shell. nt: N
o n te
Wondering what 017 C
the destroywor l d 2
VM
option does?
Better try it on
a test vm.

#SER1289BE CONFIDENTIAL 20
Tip: Location of VMware vCenter Server 6.5 Log Files

The VMware vCenter Server The VMware vCenter Server

t i o n
6.5 logs are located in the Appliance 6.5 logs are
t r i b u
%ALLUSERSPROFILE%\VM located in the r d is
o
Ware\vCenterServer\logs ion
/var/log/vmware tfolder.
c a
b l i
folder.
r pu
o t fo
n t : N
ont e
1 7 C
r ld 20Recommendation
Mw o
V
See http://kb.vmware.com/kb/2110014 and 1021806
for additional details.

#SER1289BE CONFIDENTIAL 21
Useful ESXi Host Logs for Troubleshooting

ESXi hosts write to multiple log files,

t i o n
ibu
depending on which action is being performed.
i s tr
or d
Log file a t i on
Purpose
p u blic
hostd.log
r
Host Management service logs
fo
N o t
Management service initialization, watchdogs, scheduled
syslog.log
n tent:tasks, and DCUI use
C o
17
Core VMkernel logs, including device discovery, storage
2 0
vmkernel.log and networking device and driver events, and virtual
o r ld machine startups
VMw vmkwarning.log A summary of warning and alert log messages excerpted
from the VMKernel logs
A summary of ESXi host startup and shutdown, and an
vmksummary.log hourly heartbeat with uptime, number of virtual machines
running, and service resource consumption

#SER1289BE CONFIDENTIAL 22
Tip: Other Critical Logs to Be Aware Of and how to view them

/var/log/fdm.log displays HA /var/log/shell.log displays

o n
/var/log/clomd.log displays vSAN
ti information for
cluster related events and commands executed inside b u
health irelated
tr among the output
notifications. of the esxi shell. r d i s
devices
on o
c a t i
u bli
or p
o t f
e n t:N
on t
1 7 C
r ld 20
Mw o
V

#SER1289BE CONFIDENTIAL 23
Tip: Remember /var/log

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
Lots of logs available in
nt: N
this directory
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 24
Collecting Diagnostic Data for
VMware Technical Support
t i o n
Methods for collecting diagnostic information
i s tr ibu
to send to VMware Technical Support include
or d
the following: t ion
bli c a
• Use the GUI to export files to a log bundle.
r p u
− vSphere Client or vSphere Web Client o t fo
nt: N
• Use the nte to collect
vm-support command
o
2 17 C ESXi host.
information from an individual
0
orl d
VMw

#SER1289BE CONFIDENTIAL 25
Tip: Running vm-support when engaging support

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M

#SER1289BE CONFIDENTIAL 26
 t i o n
Network Troubleshooting i s tr ibu
or d
Commands t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Tip: Useful Networking Related Commands

Use esxcfg-vmknic –l or Use esxcfg-vswitch –l or n

u t i o
vicfg-vmknic -l to view your vicfg-vswitch –l to view virtual
t r i b
vmkernel port related switch related information. rIsdis
o n o
information. Are they still there the switch connected
c a t i to at
and enabled? ub l
least one uplink? i
o r p
N otf
ent :
on t
7 C
Use esxcfg-nics –l or1vicfg- esxcli also has options to
2 0
o ld related
nics –l to view ruplink perform these tasks.
V MwAre the nics up or
information.
down? For example:
esxcli network nic list is similar
to esxcfg-nics -l

#SER1289BE CONFIDENTIAL 28
Tip: Did You Know That…

Somebody could accidentally If so, type esxcli network nic up t i o n

t r i b u
down a nic? –n vmnic2. is
r d
on o
c a t i
r p ubli
tf o
: N o
ont ent
17 C
d 2 0
orl
VMw

#SER1289BE CONFIDENTIAL 29
Tip: Did You Know That…

t i o n
Somebody could disable a vmkernel port If so, type esxcfg-vmknic -e to re-
t r i b u
with commands like esxcfg-vmknic -D? is
enable the vmkernel port.
r d
on o
c a t i
r p ubli
tf o
: N o
on t ent
17 C
d 2 0
orl
VMw

#SER1289BE CONFIDENTIAL 30
Tip: Rebuilding the Management Network from scratch

Somebody could accidentally t i o n

disconnect an uplink from a i s tr ibu
or d
virtual switch?
t ion
bli c a
r p u
o t fo
t: N
nffaaa
o n te
1 7 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 31
Need a packet sniffer? Remember pktcap-uw

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
There is a built-in packet o t fo
nt: N
sniffer called pktcap-uw
o n te
and an older one called
17 C
tcpdump-uw. 2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 32
 t i o n
i s tr ibu
or d
Storage Troubleshooting t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
Tip: Did You Know That…

You can see how many dastastores exist and how full
t i o n
the datastores are? This command displays vmfs, nfs,
i s tr ibu
vsan and vvol datastores. There is also a vdf –h or d
t ion
command.
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 34
Tip: Did You Know That…

t i o n
i s tr ibu
or d
t ion
You can easily view all
bli c a
sorts of information
r p u
with esxcli?
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 35
Tip: Use esxcli Storage Core Device Smart for SSD information

vSphere 6.5
Includes t i o n
i s tr ibu
Use this command to
or d
t ion
retrieve data about a
bli c a
specific SSD device. r p u
o t fo
Critical for vSAN,Flash
nt: N
Read Cache or any o n te
17 C
datastore using SSDs. 2 0
w o rld
V M

36 36
Tip: Be familiar with partedUtil

t i o n
i s tr ibu
or d
partedUtil is the command
t ion
that partitions luns. It can also
bli c a
r p u
create, resize and delete
o t fo
partitions.
nt: N
o n te
1 7 C
2 0
w o rld
V M

37
Tip: Suspect metadata corruption?

t i o n
i s tr ibu
voma is the on-disk metadata or d
t ion
analyzer for vmfs datastores.
bli c a
r p u
o t fo
nt: N
o n te
1 7 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 38
 t i o n
Troubleshooting Clusters and i s tr ibu
o r d
Performance t io n
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
vSphere HA
A reliable network connection between the hosts and VMware® vCenter Server™
is essential for enabling vSphere HA.
t i o n
i s t r ibu
or d
a t i on Heartbeat

u blic Datastores

or p
o t f
:N
tent
FDM FDM FDM

C n
ovpxa
17
vpxa hostd hostd vpxa hostd

d 2 0
orl
ESXi host (slave) ESXi host (slave) ESXi host (master)

VMw Management
Network
vCenter vpxd
Server

40
Tip: Remember F.D.M

t i o n
There is a log for HA that
i s tr ibu
you can’t see with the Web or d
t ion
or vSphere Client?
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 41
Tip: Did You Know That…

You can stop, start or restart the HA service from the cli if needed?

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 42
Tip: You can restart all services at once if needed …

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
Remember services.sh restart
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 43
Tip: Did You Know That…

esxtop is an amazing tool for esxtop has options for CPU esxtop has annoption for vSAN
u t i o
performance troubleshooting? (c), RAM (m), Disk (d,u,v)
i s tib
(-x option)?
r
and Network (n) related
or d
statistics? tion
l i c a
pu b
for
N o t
e n t :
C ont
1 7
r ld 20
Mw o
V

#SER1289BE CONFIDENTIAL 44
Tip: Did You Know That…

t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
esxtop has built-in help?
nt: N
o n te
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 45
Tip: Did You Know That…

t i o n
i s tr ibu
or d
You can add more
t ion
output to esxtop by bli c a
r p u
pressing “f”?
o t fo
nt: N
For example, you can o n te
7 C
add NUMA related
information (G Option)orld
201
VMw

#SER1289BE CONFIDENTIAL 46
Tip: Learn the vsish command and its capabilities

t i o n
vsish has hundreds of i s tr ibu
or d
subcommands.
t ion
bli c a
Just type the command and r p u
take a look … o t fo
nt: N
o n te
You can type commands like 7
1 C
2 0
cd to change directories
o r ld and
Mw
get to view information.
V

#SER1289BE CONFIDENTIAL 47
Tip: Learn the vsish command and its capabilities

t i o n
vsish has hundreds of i s tr ibu
or d
subcommands.
t ion
bli c a
r p u
o t fo
nt: N
Just type the command and
o n te
take a look …
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 48
Additional Education Resources
At VMworld 2017 Europe
• Education & Certification Lounge: VMvillage, Hall 6
t i o n
• Certification Exam Center: CC5
t r i b uSave 50%
r dis
on o off VCP & VCAP
c a t i
ubli
Online exams at
o r p VMworld 2017
• VMware Training: www.vmware.com/education
N o tf
• VMware Certification: www.vmware.com/certification
ent :
on t
17 C
d 2 0
w o
Global Support Services rl
VM
Learn more about how VMware is radically transforming
Customer Support through VMware Skyline™ technology.
• Visit the demo booth within the Solutions Exchange
• Sign up for a Meet the Experts session in the Content Catalogue
• Visit www.vmware.com/support/service/skyline

#SER1289BE CONFIDENTIAL 49
Prove Your Expertise with VMware Digital Badges
What are VMware Digital Badges? NEW!
VMware Digital Badges
• Single source that combines your credential and
a complete overview of your skills
t i o n
• Way for you to easily share your accomplishments
VMware vSAN 2017
t rib u
d i
Specialist
r s
in social media
o n o
• Provides employers with easy, valid verification of
u bli catiProve your expertise
VMware credentials
or p on vSAN and
o t f vSAN HCI environments

e n t:N
Digital Badges Online on t
1 7 C
• 2
vmware.com/go/vSAN2017badge
d 0
• w o rl
VM
vmware.com/go/VxRail2017badge
Dell EMC VMware
Co-Skilled - VxRail 2017

#SER1289BE CONFIDENTIAL 50
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M
 t i o n
i s tr ibu
or d
t ion
bli c a
r p u
o t fo
nt: N
o n te
17 C
2 0
w orld
V M