ser1289bu vsphere troubleshooting tips and … menendez & francis daly ser1289bu vsphere...
Post on 17-Mar-2018
253 Views
Preview:
TRANSCRIPT
Javier Menendez & Francis Daly
SER1289BU
vSphere Troubleshooting Tips and Tricks
#VMworld #SER1289BU
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1Use of the VMware vSphere Command-line Interface (vCLI),
the ESXi Shell, and Log Files to Diagnose and Correct Problems
2 Troubleshooting Networking Issues
3 Troubleshooting Storage Issues
4 Troubleshooting vSphere HA Cluster and Performance Issues
2#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
3Certain troubleshooting topics are addressed in the VMware vSphere: Fast Track [6.5] course.
Scope
This Presentation covers portions of the Troubleshooting Workshop:
For more information, go to http://vmware.com/education
VMworld 2017 Content: Not fo
r publication or distri
bution
Troubleshooting Process
#SER1289BU CONFIDENTIAL 5
The troubleshooting process consists of the following tasks:
Defining the problem
• Identifying symptoms
• Gathering information
Identifying the cause of the problem
• Identifying possible causes
• Determining the root cause
Solving the problem
• Identifying possible solutions
• Implementing the best solution
VMworld 2017 Content: Not fo
r publication or distri
bution
Solving the Problem … Can I Do It with the Web Client or Should I Use the esxi Shell?
#SER1289BU CONFIDENTIAL 6
What are my choices:
• Can I fix the problem with the Web, Host or vSphere Client?
• Do I want or need to use the command line utilities?VMworld 2017 Content: N
ot for publicatio
n or distribution
Command-line Troubleshooting Choices
#SER1289BU CONFIDENTIAL 8
Choices to obtain command-line access on a VMware® ESXi™ host:
vSphere Management Assistant (VMA)
VMware vSphere® ESXi™ Shell
• Can be access via:
• 1. The DCUI (if enabled via ALT + F1)
• 2. Via putty after enabling ssh.
• Appliance deployed via ovf
• Contains vCLI package
• Does NOT use port 22.VMworld 2017 Content: N
ot for publicatio
n or distribution
The esxcli Command
#SER1289BU CONFIDENTIAL 9
The esxcli command
offers the following
namespaces,
as well as many new esxcli functions added
in the vSphere 6.5.
If working with vSAN,
check out the new esxcli
vsan debug namespaceVMworld 2017 Content: N
ot for publicatio
n or distribution
Example: The esxcli vsan Namespace
#SER1289BU CONFIDENTIAL 10
The vSAN related
commands are found in
esxcli vsan
VMworld 2017 Content: Not fo
r publication or distri
bution
New Commands Added with Every New vSphere Release. Ex: The esxcli vsan debug Namespace
#SER1289BU CONFIDENTIAL 11
The new vSAN debug related commands are amazing to discover the state of your objects, vms, etc.
VMworld 2017 Content: Not fo
r publication or distri
bution
esxcli esxcli command list
displays all the command
options.
To quickly find the command
needed, combine it with the
grep command
Tip: Remember the esxcli esxcli command list Command
#SER1289BU CONFIDENTIAL 12
VMworld 2017 Content: Not fo
r publication or distri
bution
The VMA’s vicfg-* Commands
#SER1289BU CONFIDENTIAL 13
vicfg- [tab] [tab]
• Commands with the vicfg- prefix enable you to manage
your storage, network, and host configuration.
• Type vicfg- followed by the TAB key twice so see all the available commands.
• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.
VMworld 2017 Content: Not fo
r publication or distri
bution
The esxi shell’s esxcfg- Commands Are Still There
#SER1289BU CONFIDENTIAL 14
esxcfg- [tab] [tab]
• Commands with the esxcfg-
prefix enable you to manage
your storage, network, and
host configuration.
• Type esxcfg- followed by the TAB key twice so see all the available commands.
• Remember the –h (help) option to master these commands.
VMworld 2017 Content: Not fo
r publication or distri
bution
Use the –h option (for help)
to see options for this
particular command and in
some cases, even
examples on how to use the
command.
How many times did you
struggle to create a
vmkernel port with the cli?
Tip: Remember the “-h” Option for esxcfg-* Commands
#SER1289BU CONFIDENTIAL 15
VMworld 2017 Content: Not fo
r publication or distri
bution
Manipulating Virtual Machines with the cli from the V.M.A.?
#SER1289BU CONFIDENTIAL 16
• vmware-cmd
– The vmware-cmd command is exclusively used for virtual machines.
vmware-cmd
• The vmware-cmd command is used to interact with virtual machines.
VMworld 2017 Content: Not fo
r publication or distri
bution
Example: Listing Your Virtual Machines
#SER1289BU CONFIDENTIAL 17
vmware-cmd -l
• Lists the virtual machines that are located on the target host. Lists virtual machines by path to the .vmx file.
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn More on Your Own
18
Type vmware-cmd --help to
see what the command allows
you to do.
vmware-cmd allows you to
start a vm, take snapshots,
register a vm, etc.
The esxi host has a similar
command to manipulate vms
called vim-cmd vmsvc.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Manipulating Virtual Machines from the esxi shell
19
vim-cmd vmsvc can
perform similar
tasks from the
esxi shell.
Wondering what
the destroy
option does?
Better try it on
a test vm.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Location of VMware vCenter Server 6.5 Log Files
#SER1289BU CONFIDENTIAL 20
The VMware vCenter Server
6.5 logs are located in the
%ALLUSERSPROFILE%\VM
Ware\vCenterServer\logs
folder.
The VMware vCenter Server
Appliance 6.5 logs are
located in the
/var/log/vmware folder.
Recommendation
See http://kb.vmware.com/kb/2110014 and 1021806 for additional details.
VMworld 2017 Content: Not fo
r publication or distri
bution
Useful ESXi Host Logs for Troubleshooting
ESXi hosts write to multiple log files, depending on which action is being performed.
21
Log file Purpose
hostd.log Host Management service logs
syslog.logManagement service initialization, watchdogs, scheduled
tasks, and DCUI use
vmkernel.log
Core VMkernel logs, including device discovery, storage
and networking device and driver events, and virtual
machine startups
vmkwarning.logA summary of warning and alert log messages excerpted
from the VMKernel logs
vmksummary.log
A summary of ESXi host startup and shutdown, and an
hourly heartbeat with uptime, number of virtual machines
running, and service resource consumption
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Other Critical Logs to Be Aware of and How to View Them
22
/var/log/fdm.log displays HA
cluster related events and
notifications.
/var/log/shell.log displays
commands executed inside
of the esxi shell.
/var/log/clomd.log displays vSAN
health related information for
devices among the output
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Remember /var/log
#SER1289BU CONFIDENTIAL 23
Lots of logs available in
this directory
VMworld 2017 Content: Not fo
r publication or distri
bution
Collecting Diagnostic Data for
VMware Technical Support
24
Methods for collecting diagnostic information to send to VMware Technical Support include the following:
• Use the GUI to export files to a log bundle.
− vSphere Client or vSphere Web Client
• Use the vm-support command to collect
information from an individual ESXi host.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Running vm-support when Engaging Support
25#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Useful Networking Related Commands
27
Use esxcfg-vmknic –l or
vicfg-vmknic -l to view your
vmkernel port related
information. Are they still there
and enabled?
Use esxcfg-vswitch –l or
vicfg-vswitch –l to view virtual
switch related information. Is
the switch connected to at
least one uplink?
Use esxcfg-nics –l or vicfg-
nics –l to view uplink related
information. Are the nics up or
down?
esxcli also has options to
perform these tasks.
For example:
esxcli network nic list is similar
to esxcfg-nics -l
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could accidentally
down a nic?
If so, type esxcli network nic up
–n vmnic2.
Tip: Did You Know That…
#SER1289BU CONFIDENTIAL 28
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could disable a vmkernel port
with commands like esxcfg-vmknic -D?
If so, type esxcfg-vmknic -e to re-
enable the vmkernel port.
Tip: Did You Know That…
#SER1289BU CONFIDENTIAL 29
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could accidentally
disconnect an uplink from a
virtual switch?
ffaaa
Tip: Rebuilding the Management Network from Scratch
#SER1289BU CONFIDENTIAL 30
VMworld 2017 Content: Not fo
r publication or distri
bution
There is a built-in packet
sniffer called pktcap-uw
and an older one called
tcpdump-uw.
Need a Packet Sniffer? Remember pktcap-uw
#SER1289BU CONFIDENTIAL 31
VMworld 2017 Content: Not fo
r publication or distri
bution
You can see how many dastastores exist and how full
the datastores are? This command displays vmfs, nfs,
vsan and vvol datastores. There is also a vdf –h
command.
Tip: Did You Know That…
#SER1289BU CONFIDENTIAL 33
VMworld 2017 Content: Not fo
r publication or distri
bution
You can easily view all
sorts of information
with esxcli?
Tip: Did You Know That…
#SER1289BU CONFIDENTIAL 34
VMworld 2017 Content: Not fo
r publication or distri
bution
Use this command to
retrieve data about a
specific SSD device.
Critical for vSAN, Flash
Read Cache or any
datastore using SSDs.
vSphere 6.5
Includes
Tip: Use esxcli Storage Core Device Smart for SSD Information
35 35#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Be Familiar with partedUtil
36
partedUtil is the command
that partitions luns. It can also
create, resize and delete
partitions.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Suspect Metadata Corruption?
#SER1289BU CONFIDENTIAL 37
voma is the on-disk metadata
analyzer for vmfs datastores.
VMworld 2017 Content: Not fo
r publication or distri
bution
vSphere HA
39
vCenter
Server
ESXi host (slave)
FDM
ESXi host (master)
FDM
ESXi host (slave)
FDM
vpxd
hostdhostdhostd
Management
Network
vpxa vpxa vpxa
Heartbeat
Datastores
A reliable network connection between the hosts and VMware® vCenter Server™
is essential for enabling vSphere HA.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
There is a log for HA that
you can’t see with the Web
or vSphere Client?
Tip: Remember F.D.M
40#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
You can stop, start or restart the HA service from the cli if needed?
Tip: Did You Know That…
41#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: You Can Restart All Services at Once If Needed …
42
Remember services.sh restart
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Did You Know That…
43
esxtop is an amazing tool for
performance troubleshooting?esxtop has options for CPU
(c), RAM (m), Disk (d,u,v)
and Network (n) related
statistics?
esxtop has an option for vSAN
(-x option)?
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
esxtop has built-in help?
Tip: Did You Know That…
44#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
You can add more
output to esxtop by
pressing “f”?
For example, you can
add NUMA related
information (G Option)
Tip: Did You Know That…
45#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn the vsish Command and Its Capabilities
46
vsish has hundreds of
subcommands.
Just type the command and
take a look …
You can type commands like
cd to change directories and
get to view information.
#SER1289BU CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn the vsish Command and Its Capabilities
vsish has hundreds of
subcommands.
Just type the command and
take a look …
#SER1289BU CONFIDENTIAL 47
VMworld 2017 Content: Not fo
r publication or distri
bution
Global Support Services
Learn more about how VMware is radically transforming Customer Support through VMware Skyline™ technology.
• See a demo in the VMware booth in the Solutions Exchange
• Sign up for a Meet the Experts roundtable in the Schedule Builder on the VMworld mobile app or visit the Meet the Experts, Level 2
• Visit www.vmware.com/support/service/skyline
#SER1289BU CONFIDENTIAL 48
VMworld 2017 Content: Not fo
r publication or distri
bution
Additional Education Resources
At VMworld 2017
• Education & Certification Lounge: VM Village
• Certification Exam Center: Jasmine EFG, Level 3
Online
• vSphere Training: www.vmware.com/go/vsphere65training
• VMware Training: www.vmware.com/education
• VMware Certification: www.vmware.com/certification
#SER1289BU CONFIDENTIAL 49
Save 50% off VCP & VCAP
exams at
VMworld 2017
VMworld 2017 Content: Not fo
r publication or distri
bution
top related